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The  U.S.  Amy  Research  Institute  for  the  Behavioral  and 
Social  Sciences  perfoms  research  and  develops  programs  to 
improve  training  effectiveness  and  to  contribute  to  training 
readiness.  Of  special  Interest  are  research  and  development 
programs  that  apply  computers  and  other  advanced  technologies  to 
part>task  trainers  and  training  strategies.  Research  that  iden¬ 
tifies  the  most  effective  strategies  for  designing  computer- 
based  trainers  and  training  programs  will  enhance  the  development 
and  procurement  of  specific  part-task  training  systems  required 
by  the  Amy  training  community. 

This  report  surveys  the  literature  and  evaluates  the  exist¬ 
ing  research  for  computer-based  instructional  design  strategies. 
Instructional  design  Issues  requiring  further  research  and  devel¬ 
opment  are  identified.  With  an  understanding  of  the  current 
state  of  the  art  in  instmctlonal  design,  training  developers 
(Directorate  of  Training  and  Doctrine  (DOTD) ,  U.S.  Amy  Aviation 
Center  (USAAVNC) )  can  design  new  systems  to  use  the  most  effec¬ 
tive  strategies  and  evaluate  the  effectiveness  of  new  strategies. 

This  work  was  conducted  within  the  Training  Research  Labora¬ 
tory  program  under  Research  Task  3309,  entitled  "Techniques  for 
Tactical  Flight  Training."  The  Amy  Research  Institute  Aviation 
Research  and  Development  Activity  (ARIARDA)  at  Fort  Rucker,  Ala¬ 
bama,  was  responsible  for  the  execution  of  the  work.  This  work 
is  a  part  of  a  Memorandum  of  Agreement  (MOA)  between  ARIARDA  and 
DOTD,  USAAVNC,  dated  15  March  1984  and  updated  in  October  1989. 
This  infomatlon  was  provided  as  a  preliminary  draft  to  the  DOTD. 
However,  the  primary  application  was  a  literature  review  to  sup¬ 
port  research  for  the  empirical  development  of  a  total  training 
system  for  aviation  knowledge  and  skills. 
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STRATEGIES  OF  COMPUTER-BASED  INSTRUCTIONAL  DESIGN;  A  REVIEW 
OF  GUIDELINES  AND  EMPIRICAL  RESEARCH 


EXECUTIVE  SUMMARY 


Requirement : 

This  literature  survey  was  conducted  to  examine  the  empiri¬ 
cal  research  support  for  the  numerous  guidelines  and  recommenda¬ 
tions  that  have  been  published  about  computer-based  instructional 
design  strategies. 


Procedure : 

Two  types  of  publications  were  selected  and  reviewed: 

(a)  reports  of  empirical  research  comparing  at  least  one  strategy 
of  computer-based  Instruction  (CBI)  with  another,  and  (b)  pub¬ 
lished  guidelines  for  the  development  of  CBI.  The  guidelines 
and  experiments  were  categorized  according  to  whether  they  per¬ 
tained  to  strategies  for  (a)  presenting  instructional  material, 

(b)  questioning  and  interactivity,  or  (c)  programming  response 
feedback  and  remediation  procedures.  Each  section  of  the  report 
begins  with  a  list  of  guidelines  for  some  aspect  of  instructional 
design  and  a  discussion  of  the  recommendations  made  by  the 
authors  of  the  guidelines.  Then  the  empirical  research  relevant 
to  each  guideline  is  reviewed,  and  guidelines  that  lack  research 
support  are  identified.  Finally,  each  guideline  discussed  in  the 
report  is  classified  by  whether  (a)  authors  of  instructional 
guidelines  made  another  recommendation  that  contradicted  the 
guideline,  and  (b)  empirical  research  supports  the  guideline, 
contradicts  it,  or  does  not  exist  to  evaluate  it. 


Findings: 

Only  5  of  the  57  guidelines  reviewed  are  supported  by  em¬ 
pirical  research.  Authors  of  guidelines  agree  that  CBI  should 
present  questions,  corrective  feedback  for  Incorrect  responses, 
and  multiple  trials  for  items  that  are  answered  incorrectly. 
Although  some  authors  do  not  agree  that  CBI  should  present  pre¬ 
lesson  questions  or  that  computers  should  be  programmed  to  con¬ 
trol  adaptively  the  number  of  trials,  empirical  research  suggests 
that  these  strategies  are  effective. 

Although  the  experimental  evidence  is  not  strong  enough  to 
justify  complete  rejection,  8  of  the  57  guidelines  are  contra¬ 
dicted  by  the  results  of  empirical  research.  Empirical  research 
tends  to  contradict  recommendations  to  use  graphics  often,  to 
provide  increasingly  informative  feedback  for  successive  errors, 
to  train  under  mild  speed  stress,  to  present  information  before 


practice,  to  randomize  the  sequence  of  material,  to  present  part- 
task  training  prior  to  whole-task  training,  to  permit  the  student 
to  control  the  presentation  of  reviews,  and  to  program  the  com¬ 
puter  to  control  the  presentation  of  reviews. 

Finally,  44  of  the  guidelines  have  been  insufficiently 
addressed  by  empirical  research.  For  many  of  the  guidelines, 
either  empirical  research  has  produced  mixed  results  and  further 
research  is  required,  is  inconclusive  because  of  inadequate  ex¬ 
perimental  designs,  or  simply  does  not  exist.  Some  of  the  guide¬ 
lines,  however,  are  self-evident  and  do  not  require  empirical 
research.  Other  guidelines  are  stated  in  terms  that  are  too 
general  to  evaluate  their  usefulness.  Some  are  very  specific, 
and  an  evaluation  of  the  possibility  of  presenting  them  generally 
is  required. 


Utilization  of  Findings: 

The  CBI  designer  and  the  educational  technology  researcher 
should  be  aware  of  the  contradictions  and  potential  limitations 
of  the  recommendations  in  the  literature.  Research  should  be 
designed  to  address  the  current  gaps  in  knowledge  of  computer- 
based  instructional  strategies.  Existing  operational  problems  in 
part-task  training  may  be  addressed  in  a  manner  that  extends  this 
knowledge.  Instructional  designers  currently  working  on  training 
development  should  recognize  when  a  particular  strategy  is  being 
applied  in  a  case  that  potentially  tests  the  generality  of  a 
recommendation  or  research  result. 
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STRATEGIES  OF  COMPUTER-BASED  INSTRUCTIONAL  DESIGN: 
A  REVIEW  OF  GUIDELINES  AND  EMPIRICAL  RESEARCH 


Introduction 


Background 

Training  technology  has  progressed  from  Pressey’s 
teaching  machines  and  Link’s  simulators  of  the  1920s  to  the 
computer-based  tutors,  part-task  trainers  and  simulators  of 
the  1980s  (Benjamin,  1988;  Flexman  &  Stark,  1987) .  A  new 
field  of  research  and  development,  called  instructional 
design,  has  emerged  in  industry  and  academia  to  provide 
services  in  educational  technology.  Textbooks  on  the  appli¬ 
cation  of  learning  theory  to  instructional  design  (for 
example,  Gagne,  Briggs,  &  Wager,  1988),  journals  specializing 
in  the  publication  of  educational  technology  research  (for 
example.  Journal  of  Computer-based  Instruction,  Journal  of 
Instructional  Development) ,  ana  guidelines  for  the  develop¬ 
ment  of  computer-based  instruction  (for  example,  Alessi  & 
Trollip,  1985;  Kearsley,  1986)  have  been  published.  Profes¬ 
sional  organizations,  such  as  the  Association  for  the 
Development  of  Computer-Based  Instructional  Systems  and  the 
Computer  Systems  Technical  Group  of  the  Human  Factors 
Society,  convene  regularly  to  discuss  developments  in 
computer-based  instruction  research. 

Computer-based  instruction  (CBI)  and  simulation  are 
rapidly  becoming  an  integral  part  of  military  training. 
Operation  of  sophisticated  weapons  systems  during  fast-paced 
combat  requires  a  multitude  of  skills  that  may  be  trained 
most  effectively  with  simulators  and  part-task  trainers.  The 
development  of  such  training  technology  requires  a  thorough 
understanding  of  (a)  the  skills  and  abilities  required  to 
operate  the  systems,  (b)  theories  of  learning  and  principles 
of  instruction,  and  (c)  the  state-of-the-art  in  computer 
technology . 

Over  the  past  decade,  an  enormous  amount  of  research  has 
been  conducted  to  investigate  the  training  potential  of  CBI. 
The  majority  of  this  research  has  had  as  its  primary  objec¬ 
tive  the  comparison  of  CBI  with  other  methods  of  instruction. 
Reviewers  have  often  concluded  that  computers  can  measurably 
improve  training  effectiveness  (for  example,  Eberts  &  Brock, 
1987) .  However,  Clark  (1985)  suggests  that  the  training 
effectiveness  of  CBI  is  due  primarily  to  effective  instruc¬ 
tional  strategies  and  not  to  the  medium  per  se .  Most  of  the 
media  comparison  research  provides  very  little  information 
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about  effective  strategies  of  computer-based  instructional 
design  (Reeves,  1986;  Shlechter,  1986) . 


Procedure 

The  present  survey  of  literature  was  conducted  to 
examine  the  empirical  research  support  for  the  numerous 
guidelines  and  recommendations  that  have  been  published  about 
computer-based  instructional  design  strategies.  Two  types  of 
publications  were  selected  and  reviewed:  (a)  reports  of 
empirical  research  comparing  at  least  one  strategy  of  CBI 
with  another  and  (b)  published  guidelines  for  the  development 
of  CBI.  Reports  of  experiments  comparing  CBI  with  other 
methods  of  instruction,  reports  of  computer-based  instruc¬ 
tional  development  (without  experimentation) ,  and  reports 
describing  new  technology  or  new  applications  of  existing 
technology  were  not  considered. 

The  guidelines  and  experiments  were  categorized 
according  to  whether  they  pertained  to  strategies  for  (a) 
presenting  instructional  material,  (b)  questioning  and 
interactivity,  or  (c)  programming  response  feedbacJc  and 
remediation  procedures.  In  the  following  sections,  the 
guidelines  and  research  in  each  of  these  areas  are  presented. 
Each  section  begins  with  a  list  of  guidelines  and  a  discus¬ 
sion  of  the  recommendations  made  by  their  authors.  Then  the 
empirical  research  relevant  to  each  guideline  is  reviewed. 
Finally,  the  guidelines  that  lack  research  support  are 
identified. 


Presentation  of  Instructional  Stimuli 

Designers  of  CBI  must  make  many  decisions  about  the 
organization  and  presentation  of  training  material.  They 
must  decide  what  sorts  of  events  should  occur  on  the  monitor, 
such  as  tutorials,  drills,  and  simulations,  and  how  long  each 
instructional  event  should  take.  They  must  determine  how 
students  should  be  instructed  to  interact  with  a  lesson,  now 
graphics  can  be  used  most  effectively,  and  whether  the  stu¬ 
dent  or  the  computer  should  control  the  training  variables. 
This  section  presents  some  of  the  guidelines  and  empirical 
research  pertaining  to  orienting  instructions,  stimulus 
display  duration,  sequencing  of  instructional  material, 
sequencing  various  levels  of  difficulty,  use  of  graphics,  and 
review  of  material. 
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Orienting  Instructions  and  Qb;iectives 

Orienting  instructions  are  directions  that  are  provided 
to  inform  students  about  the  nature  of  the  interaction 
required  during  a  lesson,  the  operation  of  a  training  device, 
or  both.  Objectives  are  statements  of  the  benefits  to  be 
realized  by  the  student  as  a  result  of  successful  participa¬ 
tion  in  the  lesson  or  training.  Not  all  recommendations 
about  the  presentation  of  orienting  instructions  or  objec¬ 
tives  must  be  supported  by  empirical  research.  If  a  training 
device  is  complicated  to  operate  or  unfamiliar  to  the 
student,  operating  instructions  obviously  are  necessary  for 
training  to  occur.  Similarly,  if  a  student  must  choose  from 
a  set  of  lessons  or  training  devices,  statements  of  objec¬ 
tives  may  enable  the  student  to  select  a  lesson  addressing  a 
particular  interest  or  training  deficiency.  However,  train¬ 
ing  effectiveness  research  is  required  to  investigate  the 
possible  benefits  of  orienting  instructions  or  objectives  in 
any  particular  lesson  or  with  a  device  that  is  not  difficult 
to  operate. 


Guidelines .  Authors  of  instructional  design  guidelines 
have  four  different,  and  somewhat  incompatible,  recommenda¬ 
tions  about  the  use  of  orienting  instructions  and  objectives. 
Statements  made  about  the  utility  of  orienting  instructions 
and  statements  of  objectives  include  the  following: 

•  they  are  ineffective  and  unnecessary, 

•  they  limit  the  amount  or  breadth  of  learning, 

•  they  should  be  included  in  CBI  design,  and 

•  they  should  be  a  student-controlled  option. 

Hannafin  and  Hughes  (1986)  suggest  that  orienting 
instructions  are  typically  unnecessary.  Although  orienting 
instructions  may  be  necessary  for  poorly  organized  lessons, 
well  organized  lessons  are  not  enhanced  by  orienting  instruc¬ 
tions.  Statements  of  behavioral  objectives  may  be  beneficial 
only  when  students  have  had  some  experience  in  the  use  of 
objectives  as  orienting  activities. 

Statements  of  objectives  may  enhance  training  described 
in  the  objectives,  but  they  may  obscure  any  incidental 
training  potential  of  a  lesson.  Hannafin  and  Hughes  (1986) 
warn  that  orienting  activities  may  be  effective  only  for 
learning  that  is  "context-bound  or  limited  in  application" 

(p.  248) ,  Jonassen  and  Hannum  (1987)  and  Wager  and  Wager 
(1985)  state  that  orienting  instructions  appear  to  increase 
the  speed  of  learning  and  reduce  lesson  errors,  but  they  may 
hinder  the  retention  of  lesson  material. 
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Despite  the  conclusions  and  recommendations  described 
above,  these  and  other  authors  still  advocate  the  use  of 
orienting  instructions  and  objectives  in  CBI .  Jonassen  and 
Hannum  (1987)  advise  instructional  designers  to  include 
cognitive  maps  and  structured  overviews  to  illustrate  where 
topics  fit  into  the  content  area.  Wager  and  Wager  (1985) 
recommend  that  lessons  inform  the  student  of  special  condi¬ 
tions  and  expectations  of  performance.  Alessi  and  Trollip 
(1985)  state  that  computer-based  tutorials  should  have  a 
concise  and  accurate  statement  of  objectives,  although  not 
necessarily  in  behavioral  terms.  However,  they  add  that  this 
recommendation  does  not  apply  to  CBI  designed  for  young 
children.  Buehner  (1987)  suggests  that  CBI  consistently 
include  an  overview  of  the  training,  describing  opportunities 
for  review  and  reinforcement.  According  to  MacLachlan 
(1986),  "...at  the  beginning  of  a  computerized  tutorial,  the 
student  can  be  told  the  benefits  he  can  expect  by  giving 
attention  to  the  material  in  the  tutorial ....  It  would  be 
desirable,  then,  to  preface  a  tutorial  with  instructions 
indicating  the  manner  in  which  the  student  should  be  watch¬ 
ful"  (p.  65) .  Finally,  Jonassen  and  Hannum  (1987)  recommend 
that  lesson  overviews  be  presented  as  an  option  for  the 
student  to  choose . 

Research .  There  is  very  little  research  support  for 
recommendations  about  orienting  instructions  and  objectives. 
Ho,  Savenye,  and  Haas  (1986)  and  Reiber  and  Hannafin  (1986) 
found  no  differences  between  groups  of  subjects  who  viewed 
orienting  instructions  and  those  who  did  not.  However, 
Hannafin,  Phillips,  and  Tripp  (1986)  found  that  orienting 
instructions  improved  posttest  scores  when  the  ensuing  lesson 
followed  the  instructions  after  a  5-second  interval  but  not 
after  a  20-second  interval. 

The  research  evidence  is  inadequate  to  support  either 
the  inclusion  or  exclusion  of  orienting  instructions  or 
objectives  in  CBI.  Furthermore,  no  research  has  been  located 
that  addresses  the  assertion  that  orienting  instructions  and 
objectives  limit  the  amount  or  breadth  of  learning  in  CBI. 


Stimulus-  Bi^Rlay  Puration 

Except  in  the  case  of  animation,  the  CBI  designer  must 
decide  whether  a  stimulus  display  should  remain  on  the  screen 
until  removed  by  the  student,  or  should  be  removed  automati¬ 
cally  after  a  prescribed  amount  of  time  has  passed.  If  the 
latter  alternative  is  chosen,  the  designer  must  determine 
optimal  durations  for  displays. 
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Ruidt^lines .  Authors  of  instructional  design  guidelines 
disagree  in  their  reconmendations  about  stimulus  display 
duration.  Alessi  and  Trollip  (1985)  advocate  student  control 
of  display  duration  for  the  presentation  of  questions  and 
graphics.  However,  Eberts  and  Brock  (1984)  assert,  "If  the 
task  can  be  shortened  with  no  loss  of  those  aspects  of  the 
task  that  are  important  for  training,  then  more  practice 
trials  can  be  achieved  in  the  same  amount  of  time"  (p.  268) . 
Shortening  a  task  may  require  computer  control,  not  student 
control,  of  display  durations.  Furthermore,  Schneider, 
Vidulich,  and  Yeh  (1982)  recommend  that  training  be  conducted 
under  mild  speed  stress;  speed  stress  is  difficult  to  achieve 
when  the  control  of  display  duration  is  left  to  the  student. 

Research .  Control  of  display  duration  was  examined  in 
four  experiments.  Tennyson  and  Park  (1984)  compared  two 
strategies  of  computer-controlled  display  durations  with  a 
condition  of  student-controlled  duration.  In  one  computer- 
control  strategy,  display  duration  increased  following 
correct  responses  and  decreased  following  incorrect 
responses.  In  the  other  computer-control  strategy,  display 
duration  decreased  following  correct  responses  and  increased 
following  incorrect  responses.  Posttest  scores  were  higher 
when  display  duration  increased  following  correct  responses 
and  decreased  following  incorrect  responses,  but  there  were 
no  significant  differences  between  the  student-control 
strategy  and  either  computer-control  strategy. 

Tennyson,  Park,  and  Christensen  (1985)  and  Tennyson, 
Welsh,  Christensen,  and  Hajovy  (1985)  each  compared  two 
strategies  for  manipulating  trial  display  duration.  In  one 
strategy,  display  duration  increased  following  correct 
responses  but  was  unchanged  following  incorrect  responses . 

In  the  other  strategy,  display  duration  was  controlled  by  the 
student .  Posttest  scores  were  higher  for  students  who  used 
the  former  strategy. 

Two  of  these  three  experiments  support  the  recommenda¬ 
tion  that  the  computer,  rather  than  the  student,  control 
display  duration.  However,  the  results  of  these  experiments 
also  suggest  that  student  control  of  display  duration  might 
be  appropriate  after  the  student  has  progressed  beyond 
initial  skill  or  knowledge  acquisition.  In  each  of  these 
experiments,  a  student's  response  immediately  terminated  the 
trial  display  and  initiated  the  feedback  procedure.  In  the 
most  effective  strategy  of  each  experiment,  correct  responses 
produced  longer  display  durations.  As  the  frequency  of  cor¬ 
rect  responses  increased,  each  subject  had  more  opportunity 
to  respond  before  a  trial  was  automatically  terminated. 
Therefore,  the  students  who  were  responding  correctly 
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essentially  controlled  their  own  trial  durations.  Tennyson, 
Park,  and  Christensen  (1985)  and  Tennyson,  Welsh, 

Christensen,  and  Hajovy  (1985)  showed  that  student  control 
was  not  an  effective  strategy  when  it  was  unrelated  to  the 
student's  correct  responding. 

All  three  experiments  contradict  the  recommendations  to 
decrease  trial  durations  or  train  under  speed  stress.  That 
is,  if  correct  responses  result  in  longer  display  durations, 
successful  performance  could  make  the  program  run  slower  and 
probably  become  less  stressful,  rather  than  quicker  and  more 
stressful . 

Finally,  one  experiment  appears  to  support  student 
control  of  display  duration.  Belland,  Taylor,  Canelos, 

Dwyer,  and  Baker  (1985)  compared  three  strategies  for 
controlling  duration  of  a  textual  display.  In  one  strategy, 
the  student  controlled  the  duration  of  each  lesson  frame.  In 
another  strategy,  the  frame  duration  equalled  one  second  per 
line  of  text  plus  eight  extra  seconds  per  frame.  In  the 
third  strategy,  the  frame  duration  equalled  one  second  per 
line  of  text  plus  one  extra  second  per  frame.  Posttest 
scores  were  higher  for  students  using  the  student-control  and 
the  one-plus-eight-second  strategy  than  for  students  using 
the  one-plus-one-second  strategy.  While  appearing  to  support 
the  student-control  recommendation,  the  results  of  the 
research  conducted  by  Belland  et  al.  (1985)  merely  indicate 
that  one  second  per  line  plus  one  second  per  frame  is  an 
insufficient  amount  of  time  to  read  text. 

The  issue  of  controlling  text  duration  may  not  require 
empirical  research.  Unless  the  goal  of  instruction  is  to 
increase  a  student ' s  reading  speed,  text  should  not  be 
presented  faster  than  it  can  be  read.  However,  if  research 
is  required,  a  better  test  of  student  control  of  text  dura¬ 
tion  would  be  to  compare  pairs  of  students  of  comparable 
reading  speed.  One  student  in  each  pair  should  control  the 
display  duration  for  the  pair,  thereby  creating  an  apparent 
computer-control  condition  for  the  other  student. 


Sequencing  Instructional  Material 

Instructional  designers  must  decide  the  sequence  in 
which  instructional  events  occur.  For  example,  lessons  could 
begin  with  presentations  of  information,  follow  with  practice 
drills  or  questions,  and  end  with  remediation.  Part  tasks 
could  be  trained  prior  to  whole  tasks.  The  same  subject 
matter  could  be  trained  in  a  tutorial  or  information-oriented 
lesson,  a  flashcard-type  drill,  or  a  simulation  or  game. 
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Guidelines .  Instructional  designers  must  decide  whether 
the  student,  the  computer,  or  the  designer  should  determine 
the  sequence  of  events.  Authors  of  instructional  design 
guidelines  offer  the  following  conflicting  recommendations: 

•  always  allow  the  student  to  control  the  secjuence  of 
instruction, 

•  never  allow  the  student  to  control  the  sequence  of 
instruction,  and 

•  conditionally  allow  the  student  to  control  the 
sequence  of  instruction. 

Kearsley  (1986)  recommends  that  the  student  control  all 
aspects  of  sequencing.  Specifically,  students  should  be  able 
to  get  to  any  lesson  or  part  of  the  program  from  a  main  menu 
and  should  be  able  to  decide  the  order  in  which  they  would 
like  to  complete  the  lessons .  However,  Kearsley  adds  that 
the  computer  should  inform  students  if  they  choose  to  skip 
important  prerequisite  information.  Eberts  and  Brock  (1984) 
advise  that  a  program  should  promote  a  strategy  of  active 
search  through  instructional  material,  and  therefore,  should 
be  designed  so  students  can  freely  access  different  parts  of 
the  material.  Buehner  (1987)  says  that  instruction  should 
include  opportunities  for  the  student  to  adjust  the  order  of 
presentation.  Alessi  and  Trollip  (1985)  recommend  that 
designers  provide  the  capability  to  browse  through  the 
questions  in  a  computer-based  test. 

Jonas sen  and  Hannum  (1987)  advocate  the  opposite  posi¬ 
tion.  Claiming  it  is  not  effective  for  students  to  control 
the  sequence  of  instruction,  they  advise:  "Do  not  allow 
learners  to  determine  the  sequence  of  the  instructional 
content,"  and  "Do  not  allow  an  option  for  learners  to  skip 
examples"  (p.  14) . 

However,  Jonassen  and  Hannum  (1987)  contradict  them¬ 
selves  when  they  advise  designers  to  give  "learners  with  high 
internal  locus  of  control  the  ability  to  sequence  the  lesson 
content"  (p .  13).  Consequently,  they  suggest  that  instruc¬ 
tional  programs  include  a  test  for  internal  locus  of  control 
and  permit  students  with  a  high  degree  of  internal  locus  of 
control  to  sequence  the  lesson  content  for  themselves. 
Jonassen  and  Hannum  do  not  define  internal  locus  of  control 
in  their  article;  however,  the  phrase  was  used  originally  to 
refer  to  the  tendency  of  some  individuals  to  believe  that 
their  own  actions  are  responsible  for  the  consequences  of 
their  behavior  (Rotter,  1966).  Jonassen  and  Hannum's 
recommendation  assumes  that  students  who  believe  they  are  the 
causes  of  their  own  reinforcers  also  know  the  most  effective 
training  sequences  for  themselves.  Alessi  and  Trollip  (1985) 
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reconunend  that  control  of  sequence  be  given  to  advanced 
students  when  the  subject  matter  is  simple,  but  not  to  less 
advanced  students  and  not  if  the  subject  matter  is 
complicated , 

In  the  event  that  students  are  not  permitted  to  control 
the  sequence  of  instruction,  effective  secjuences  must  be 
identified  and  programmed  into  the  lesson.  Authors  of  design 
guidelines  recommend  the  following: 

•  information  should  be  presented  before  practice, 

•  concrete  material  should  be  presented  before  abstract 
material, 

•  chronological  ordering  of  material  should  be  used  in 
preference  to  random  sequencing, 

•  the  sequence  of  practice  material  should  be 
randomized,  and 

•  part-task  training  should  sometimes  precede  whole-task 
training . 

Alessi  and  Trollip  (1985)  assert  that  programs  are  more 
efficient  and  more  successful  when  they  begin  with  the 
presentation  of  information.  For  initial  acquisition  of 
information,  Jonassen  and  Hannum  (1987)  advocate  sequences 
that  begin  with  concrete,  real-world  experiences  and  proceed 
to  more  abstract  experiences.  MacLachlan  (1986)  advocates 
thematic  organization  of  material  and  recommends  chrono¬ 
logical  organization  as  the  most  effective  theme  for 
computer-based  tutorials.  In  contrast,  Jonassen  and  Hannum 
advise  designers  to  randomize  the  presentation  of  practice 
material  during  computer-based  drills. 

Eberts  and  Brock  (1984)  recommend  that  early  training 
should  be  mostly  part-task  training,  and  later  training 
should  contain  a  high  percentage  of  whole-task  training. 
However,  Holding  (1987)  notes  that  the  value  of  part-task 
training  depends  on  the  tradeoff  between  task  difficulty  and 
the  degree  of  interdependence  of  the  parts  of  a  task.  Large 
tasks  that  can  be  broken  down  into  relatively  independent 
parts  are  more  appropriately  trained  with  part-task  training. 

Research.  There  is  very  little  research  to  support 
recommendations  about  the  allocation  of  sequence  control  in 
CBI .  Gray  (1987)  compared  two  strategies  of  sequence  control 
in  CBI.  In  one  strategy,  students  could  choose  to  see  a 
review  of  each  trial,  but  otherwise  the  trials  were  presented 
in  a  linear  fashion.  In  the  other  strategy,  students  could 
choose  to  review  the  current  trial,  progress  to  the  next 
trial,  or  branch  anywhere  in  the  lesson.  Students  using  the 
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branching  strategy  performed  better  on  a  posttest  adminis¬ 
tered  immediately  following  the  instruction  than  the  students 
using  the  linear  strategy.  However,  there  were  no  differ¬ 
ences  between  groups  on  a  second  posttest  administered  one 
week  after  the  lesson. 

Except  for  the  results  of  the  follow-up  posttest,  this 
experiment  appears  to  support  the  recommendation  that  stu¬ 
dents  be  allowed  to  control  the  sequence  of  instruction. 
However,  the  effectiveness  of  the  linear  strategy  was  not 
adequately  assessed.  The  student-control  strategy  may  have 
been  merely  better  than  a  relatively  ineffective  strategy  of 
trial  sequencing.  This  does  not  contradict  the  possibility 
that  an  instructional  designer  could  determine  a  sequence 
that  is  superior  to  the  student-control  strategy. 

A  demonstration  that  student-control  strategy  is  more 
effective  than  an  ineffective  training  sequence  does  not 
refute  a  recommendation  that  the  computer  should  control  the 
instructional  sequence.  For  example,  Barsam  and  Simutis 
(1984)  compared  two  trial  sequencing  strategies  in  a 
computer-based  course  on  map  contour  line  interpretation.  In 
both  strategies  the  computer  presented  a  simplified  contour 
map  in  the  upper  half  of  the  screen  and  a  three-dimensional 
depiction  of  the  terrain  in  the  lower  half  of  the  screen. 

The  computer  cursor  could  be  placed  at  any  location  on  the 
map  and  rotated.  Subsequently,  the  three-dimensional  view 
corresponding  to  the  cursor's  position  and  direction  would 
appear  on  the  lower  half  of  the  screen.  In  the  student- 
control  strategy,  the  student  could  select  the  placement  and 
rotation  of  the  cursor  on  each  trial.  In  the  computer- 
control  strategy,  the  placement  and  rotation  of  the  cursor 
was  randomly  selected  on  each  trial.  Students  with  high 
spatial  ability  test  scores  in  the  student-control  condition 
had  higher  posttest  scores  than  students  with  high  spatial 
ability  test  scores  in  the  computer-control  condition. 
However,  there  were  no  differences  between  the  two  conditions 
for  students  with  medium  or  low  spatial  ability  test  scores. 

The  results  of  this  experiment  cannot  be  taken  to 
support  recommendations  for  student  control  of  trial 
selection  for  high  ability  students  or  recommendations 
against  student  control  for  low  ability  students .  As  with 
the  Gray  (1987)  research,  this  experiment  does  not  demon¬ 
strate  the  effectiveness  of  the  comparison  strategy  for  trial 
sequencing.  It  is  possible  that  a  random  sequence  of  trials 
is  an  ineffective  instructional  strategy,  and  students  of 
high  spatial  ability  were  simply  able  to  design  a  lesson  that 
was  superior  to  random  trial  presentations,  while  students  of 
low  spatial  ability  were  not. 
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No  research  was  found  to  support  Alessi  and  Trollip's 
(1985)  recommendation  that  CBI  should  present  information 
before  practice.  Lahey  (1981)  compared  three  presentation 
sequences  of  rules,  examples,  and  practice  trials  in  a 
concept  learning  task:  rule-examples-practice,  examples- 
rule-practice,  and  practice-examples-rule .  According  to 
Alessi  and  Trollip,  the  first  and  second  sequences  should  be 
more  effective  than  the  third.  However,  posttest  results 
revealed  no  differences  among  the  three  sequences. 

No  research  was  found  to  support  the  recommendations 
that  CBI  should  present  concrete  material  before  abstract 
material,  chronologically  order  the  instructional  material, 
or  randomize  the  sequence  of  practice  material.  In  fact,  the 
results  of  Barsam  and  Simutis's  (1984)  research  could  be 
taken  as  refutation  of  the  recommendation  to  randomize  the 
sequence  of  practice  material. 

Wightman  and  Lintern  (1985)  reviewed  empirical  research 
on  three  strategies  of  part-task  training:  segmentation, 
fractionation,  and  simplification.  In  segmentation,  a  task 
is  divided  temporally  or  spatially  into  subtasks  with  identi¬ 
fiable  end  points.  In  fractionation,  a  task  is  divided  into 
subtasks  that  are  normally  executed  simviltaneously .  In 
simplification,  a  difficult  task  is  made  easier  by  reducing 
the  complexity  of  stimuli  and  response  requirements. 

Although  many  of  the  experiments  that  Wightman  and  Lintern 
reviewed  did  not  employ  computer-based  instructional  media, 
the  results  typically  suggest  that  segmentation  is  an 
effective  strategy  of  part -task  training.  However,  the 
methods  of  fractionation  and  simplification  were  not 
necessarily  more  effective  than  whole-task  training.  The 
effectiveness  of  part-task  training  prior  to  whole-task 
training  appears  to  be  a  function  of  the  task  partitioning 
strategy,  an  issue  that  may  be  independent  of  computer-based 
instructional  design  considerations. 

A  program  of  research  by  Tennyson  and  associates 
demonstrated  the  effects  of  response-sensitive  trial 
sequencing  strategies  in  a  concept  classification  task  (Park 
&  Tennyson,  1980,  1986;  Tennyson,  Park,  &  Christensen,  1985; 
Tennyson,  Welsh,  Christensen,  &  Hajovy,  1985) .  In  each  trial 
of  a  concept  classification  task,  a  subject  was  required  to 
identify  the  category  to  which  a  particular  example  belongs. 
Park  and  Ten:  yson  (1980)  compared  a  response-sensitive  strat¬ 
egy  with  random  presentation  of  trials.  In  the  response- 
sensitive  strategy,  the  trial  following  each  incorrect 
response  contained  an  example  from  the  same  category  as  the 
previous  incorrect  response.  The  response-sensitive  strategy 
produced  higher  posttest  scores  with  fewer  training  trials 
than  the  random  presentation. 
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Then,  Tennyson,  Welsh,  Christensen,  and  Hajovy  (1985) 
compared  two  response-sensitive  strategies.  One  strategy  was 
identical  to  the  Park  and  Tennyson  (1980)  strategy;  the 
trial  following  each  incorrect  response  contained  an  example 
from  the  same  category  as  the  previous  incorrect  response 
(hereafter  called  the  previous  response  strategy) .  In  the 
other  strategy,  the  trial  following  each  incorrect  response 
contained  an  example  from  the  same  category  as  the  previous 
example  (hereafter  called  the  previous  example  strategy) . 
Posttest  scores  showed  no  differences  between  these  two 
strategies . 

Tennyson,  Welsh,  Christensen,  and  Hajovy  (1985)  then 
demonstrated  that  a  third  strategy  was  superior  to  the 
previous  example  strategy.  In  the  new  strategy,  incorrect 
responses  early  in  the  lesson  were  followed  by  a  trial 
containing  an  example  from  the  same  category  as  the  previous 
example,  but  incorrect  responses  later  in  the  lesson  were 
followed  by  a  trial  containing  an  example  from  the  same 
category  as  the  previous  response.  Tennyson,  Park  and 
Christensen  (1985)  affirmed  and  extended  this  finding  by 
demonstrating  the  superiority  of  the  new  strategy  over  the 
previous  response  strategy  as  well. 

Park  and  Tennyson  (1986)  found  no  differences  between 
the  previous  example  and  previous  response  strategies  on  a 
computer-based  multiple-choice  posttest  (reproducing  the 
results  of  Tennyson,  Welsh,  Christensen,  and  Hajovy,  1985)  . 
However,  subjects  using  the  previous  example  strategy  per¬ 
formed  significantly  better  than  subjects  using  the  previous 
response  strategy  on  a  paper-and-pencil  posttest  of  concept 
definitions . 

These  results  demonstrate  that  a  method  of  response- 
sensitive  (but  computer-controlled)  trial  sequencing  can 
produce  better  learning  than  random  sequences .  However,  it 
is  unclear  if  the  computer-control  method  is  superior  to 
student  control  of  trial  sequencing.  Furthermore,  when 
programming  response-sensitive  sequencing  strategies  for 
other  kinds  of  tasks,  instructional  designers  will  find  very 
little  guidance  in  the  results  of  these  experiments. 


Sequencing  Levels  of  Difficulty 

For  any  given  subject  matter,  instructional  material  may 
be  organized  and  presented  at  various  levels  of  difficulty. 
The  designer  must  determine  what  aspects  of  a  task  or  subject 
matter  are  difficult  to  students  and  how  the  content  should 
be  presented  for  optimal  training  effectiveness. 
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Guidelines .  Authors  of  design  guidelines  make  several 
conflicting  recommendations  about  the  sequencing  of  various 
levels  of  difficulty  in  CBI,  including  the  following: 

•  simple  material  should  be  presented  first  and 
complicated  material  should  be  presented  later, 

•  material  that  the  student  already  knows  should  not  be 
presented, 

•  the  difficulty  level  of  material  should  not  vary 
during  the  session, 

•  the  student  should  be  allowed  to  control  the  diffi¬ 
culty  level,  and 

•  the  student  should  not  be  allowed  to  control  the 
difficulty  level. 

Most  authors  recommend  that  instructional  presentations 
begin  with  simple  material  and  progress  to  more  complex 
material.  Wheaton,  Rose,  Fingerman,  Korotkin,  and  Holding 
(1976)  assert  that  mastery  of  a  task  will  be  quicker  if 
errors  are  prevented  in  the  first  few  trials.  Eberts  and 
Brock  (1984)  agree  by  recommending  that  the  early  stages  of 
training  be  designed  so  the  accuracy  rate  of  responding  is 
relatively  high.  Wager  and  Wager  (1985)  recommend  that 
designers  sequence  material  from  simple  to  complex  and  build 
on  what  the  student  already  knows.  Jonassen  and  Hannum 
(1987)  advise  designers  to  adapt  the  context  of  more  diffi¬ 
cult  tasks  to  a  content  area  familiar  to  the  student  and  have 
the  student  progress  from  easy  to  more  difficult  problems . 
Finally,  Alessi  and  Trollip  (1985)  state  that  computer-based 
tutorials  should  show  students  how  to  relate  new  information 
to  what  they  already  know. 

These  recommendations  conflict  with  MacLachlan ' s  (1986) 
advice  to  avoid  the  presentation  of  "information  that  stu¬ 
dents  already  know"  and  "provide  a  means  of  branching  around 
information  which  is  not  needed  by  certain  students"  (p.  67) . 
Similarly,  the  advice  of  Alessi  and  Trollip  (1985)  to  keep 
item  difficulty  fairly  constant  throughout  a  computer-based 
drill  session  conflicts  with  recommendations  to  progress  from 
easy  to  difficult  problems.  Ensuring  errorless  performance 
may  be  incompatible  with  keeping  item  difficulty  constant 
throughout  a  session. 

Kearsley  (1986)  advises  designers  to  let  the  student 
determine  the  difficulty  level  of  a  presentation.  However, 
other  authors  of  design  guidelines  disagree.  Asserting  that 
student  control  of  difficulty  level  is  ineffective,  Eberts 
and  Brock  (1984)  recommend  that  the  computer  be  programmed  to 
determine  the  appropriate  level  of  difficulty.  Similarly, 
Jonassen  and  Hannum  (1987)  recommend  the  use  of  courseware 
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that  analyzes  each  student's  responses  and  adjusts  the 
difficulty  level  accordingly. 

Research.  None  of  the  recommendations  about  the 
sequencing  of  different  levels  of  difficulty  are  supported  by 
computer-based  instructional  strategies  research.  Only  one 
report  was  located  that  addresses  the  organization  of 
material  according  to  difficulty  level.  Tennyson,  Welsh, 
Christensen,  and  Hajovy  (1985)  compared  two  types  of  content 
organization:  taxonomic  and  schematic.  These  authors  define 

a  taxonomic  structure  as  the  organization  of  lesson  content 
that  might  be  recommended  by  a  subject  matter  expert .  In  a 
schematic  structure,  the  lesson  content  is  organized  into  a 
hierarchy  of  prerequisite  skills  and  knowledge.  An  initial 
experiment  revealed  no  differences  between  the  two  content 
organization  strategies.  In  a  second  experiment,  preliminary 
training  in  the  operation  of  the  computer  was  added  to  each 
content  organization  condition.  Results  of  the  second 
experiment  demonstrated  the  superiority  of  schematic  over 
taxonomic  structure.  This  is  weak  support  for  the  recommen¬ 
dation  to  present  easy  material  first  and  difficult  material 
later . 


Graphics 

Many  instructional  objectives  can  be  met  through  the 
presentation  of  textual  material.  In  some  cases,  however, 
text  alone  may  be  insufficient  to  achieve  the  training 
objectives;  additional  nontextual  graphics  may  be  required. 

In  other  cases,  graphics  may  enhance  training  effectiveness 
by  illustrating  information  more  clearly  and  quickly  than 
comparable  textual  presentations.  Instructional  designers 
should  understand  how  computer  graphics  modify  the  training 
effectiveness  of  CBI.  Such  an  understanding  may  support 
decisions  about  when  graphics  should  be  used  and  what  sorts 
of  graphics  should  be  used  to  achieve  particular  training 
objectives . 

Guidelines .  Authors  of  instructional  design  guidelines 
are  more  consistent  in  their  recommendations  about  the  use  of 
graphics  than  in  other  areas  of  computer-based  instructional 
design.  They  offer  the  following  guidelines: 

•  graphics  should  be  used  often, 

•  graphics  should  be  used  for  highlighting  important 
material, 

•  graphic  information  should  be  placed  on  the  same 
screen  as,  and  to  the  left  of,  textual  material. 
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•  high  quality  graphics  are  usually  unnecessary  and 
should  be  avoided,  and 

•  animation  should  be  used. 

Several  authors  advise  designers  to  use  graphics  rather 
than  text  whenever  possible  (Kearsley,  1986;  Kearsley  & 

Frost,  1985) .  Noting  that  illustrations  and  pictures  are 
remembered  more  easily  than  words,  MacLachlan  (1986)  recom¬ 
mends  that  designers  use  concrete  rather  than  abstract  repre¬ 
sentations.  Braden  (1986)  advocates  "a  balance  between 
iconic  and  digital  displays,  appealing  to  both  right  and  left 
brain  hemispheres"  (p.  22).  Buehner  (1987)  recommends  a 
combination  of  verbal  (e.g.,  text,  voice)  and  spatial  (e.g., 
graphics,  pictures)  material  in  CBI. 

Several  authors  suggest  the  use  of  graphics  to  highlight 
important  material  (Jonassen  &  Hannum,  1987;  MacLachlan, 

1986) .  However,  Jonassen  and  Hannum  (1987)  warn  designers 
never  to  use  flashing  text  nor  more  than  three  types  of 
highlighting  cues  in  any  lesson.  Similarly,  Kearsley  and 
Frost  (1985)  warn  designers  to  use  color  sparingly  for 
emphasis.  Instead,  they  recommend  using  variations  of  type 
styles  and  sizes  to  highlight  material. 

Alessi  and  Trollip  (1985)  assert  that  graphic  informa¬ 
tion  should  be  presented  on  the  same  display  as  corresponding 
textual  information.  Wager  and  Wager  (1985)  agree,  but 
suggest  that  when  it  is  impossible  for  all  the  information  to 
be  on  the  same  screen,  the  program  should  allow  switching 
between  the  textual  and  graphic  displays.  Buehner  (1987) 
recommends  that  spatial  material  be  placed  to  the  left  of 
textual  material  to  facilitate  processing  by  the  right  hemi¬ 
sphere  of  the  brain. 

Some  authors  feel  that  excessive  detail  or  realism 
should  be  avoided  in  graphic  presentations  (Alessi  &  Trollip, 
1985)  and  that  medium  resolution  graphics  are  sufficient 
(Jonassen  &  Hannum,  1987)  .  Alessi  and  Trollip  advise 
designers  to  provide  "just  as  much  detail  as  is  necessary  to 
convey  the  necessary  information"  (p.  194),  but  give  no 
guidance  about  how  to  determine  the  optimal  level  of  detail 
for  a  specific  display.  Claiming  that  lack  of  visual  detail 
arouses  curiosity  and  enhances  training  effectiveness, 
MacLachlan  (1986)  asserts  that  "one  way  to  induce  [curiosity] 
is  to  present  the  human  subject  with  a  blurred  picture" 

(p.  65) . 

Finally,  MacLachlan  (1986)  recommends  animated  graphics 
during  tutorials.  Kearsley  (1986)  recommends  animation  to 
convey  sequence  or  cause  and  effect  relationships. 


Research.  None  of  the  research  on  the  effects  of 
graphics  in  CBI  supports  a  reconunendation  to  use  graphics . 

For  example,  Reid  and  Beveridge  (1986)  examined  the  effects 
of  illustrations  accompanying  text  in  a  computer-based 
science  lesson.  Posttest  scores  showed  no  differences 
between  subjects  who  viewed  pictures  plus  text  and  those  who 
viewed  text  only. 

Surber  and  Leader  (1988)  compared  three  types  of  feed- 
baclc  messages  in  a  computer-based  spelling  drill.  In  a 
graphic  feedback  condition,  correct  responses  were  followed 
by  a  3-second,  multicolored,  cartoon-style  display  of  a  word 
such  as  "WOW,"  "SUPER,"  or  "GOOD  WORK."  Incorrect  responses 
were  followed  by  the  word  "NO"  in  the  same  graphic  style  and 
then  the  correct  spelling  of  the  word.  In  one  of  the  non¬ 
graphic  feedback  conditions,  responses  were  followed  by  the 
same  words  as  in  the  graphic  condition;  however,  the  feedback 
appeared  as  normal  text  characters  rather  than  graphic  dis¬ 
plays.  In  the  other  nongraphic  feedback  condition,  correct 
responses  were  followed  only  by  a  presentation  of  the  next 
trial,  and  incorrect  responses  were  followed  by  a  presenta¬ 
tion  of  the  correct  spelling.  The  different  forms  of  feed¬ 
back,  and  consequently,  the  graphics,  had  no  effect  on  the 
posttest  scores  or  on  the  tendency  of  students  to  return  to 
use  the  computer  a  second  time. 

Research  support  for  the  recommendation  to  use  animation 
is  mixed.  Moore,  Nawrocki,  and  Simutis  (1979)  presented 
computer-based  lessons  on  audio  psychophysiology  to  three 
groups  of  subjects.  The  lessons  differed  among  groups  in  the 
level  of  complexity  of  graphic  displays.  For  one  group, 
graphics  consisted  of  static  alphanumeric  characters  and 
schematic  drawings.  For  another  group,  graphics  consisted  of 
static  line  drawings  of  ear  physiology.  For  the  third  group, 
graphics  consisted  of  animated  line  drawings.  No  significant 
differences  were  found  among  groups  on  any  of  four  posttests 
or  in  the  time  required  to  complete  lessons. 

Rieber  and  Hannafin  (1986)  compared  textually  presented 
orienting  instructions  with  an  animated  version  of  the  same 
instructions  and  found  no  differences.  As  noted  previously, 
research  suggests  that  orienting  instructions  may  not  enhance 
training  effectiveness.  Therefore,  Rieber  and  Hannafin 's 
research  may  not  constitute  a  valid  assessment  of  the  utility 
of  animation  in  CBI. 

Animated  graphics  may  be  more  useful  than  still  graphics 
for  training  the  perception  of  full-motion  visual  stimuli. 
McDonald  and  Whitehill  (1987)  found  that  full-motion  graphics 
had  a  more  beneficial  effect  on  posttest  performance  than 
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still  graphics  for  training  the  visual  recognition  of  moving 
stimuli . 

No  literature  was  found  that  addresses  recommendations 
about  the  placement  of  graphics  on  the  display  or  the  use  of 
graphics  for  highlighting.  Furthermore,  no  literature  was 
found  that  addresses  the  assertion  that  high  quality  graphics 
are  unnecessary  (Alessi  &  Trollip,  1985;  Jonassen  &  Hannum, 
1987/  MacLachlan,  1986) . 


Review  of  Material 

Instructional  presentations  typically  conclude  with  a 
review  of  instructional  material.  Remediation  for  an 
incorrect  answer  usually  consists  of  a  review  of  instruc¬ 
tional  segments  pertaining  to  the  test  question.  Instruc¬ 
tional  designers  are  concerned  about  whether  the  student  or 
the  computer  should  control  the  presentation  of  such  reviews. 

Guidelines .  Some  authors  of  instructional  design 
guidelines  recommend  that  students  be  allowed  to  control  the 
presentation  of  reviews.  For  example,  Cohen  (1985)  suggests 
that  CBI  programs  be  designed  to  enable  students  to  go  back 
through  a  lesson  and  reexamine  the  questions.  Alessi  and 
Trollip  (1985)  recommend  that  computer-based  tests  be 
designed  in  such  a  way  that  students  can  mark  questions  for 
subsequent  review.  Jonassen  and  Hannum  (1987)  tell  the 
designer  to  give  the  student  the  option  of  reviewing  material 
before  answering  questions. 

In  contrast,  other  authors  recommend  that  the  designer 
or  the  computer  control  the  presentation  of  reviews .  For 
example,  Jonassen  and  Hannum  (1987)  advise  designers  to 
"select  items  again  at  intervals  for  review"  (p.  10).  Wager 
and  Wager  (1985)  suggest  a  spaced  review  after  the  student 
first  masters  the  mater. ■!  .  .  Braden  (198  6)  recommends  that 
whole  displays  or  majo’  jgments  of  displays  be  repeated  to 
reinforce  learning. 

Research .  The  research  shows  no  differences  between 
student-control  and  computer-control  review  strategies.  Ho 
et  al .  (1986)  compared  three  review  strategies  during  CBI: 

computer-controlled  review,  student-controlled  review,  and  no 
review.  In  the  computer-control  strategy,  a  video  summary 
was  presented  after  each  instructional  segment.  In  the 
student-control  strategy,  each  student  chose  whether  or  not 
to  view  the  summary.  In  the  no-review  strategy,  each 
instructional  segment  was  followed  immediately  by  the  next 
instructional  segment.  Students  using  the  two  review 
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strategies  had  significantly  higher  posttest  scores  than 
students  using  the  no-review  strategy;  however,  there  were  no 
differences  between  the  computer-controlled  and  student- 
controlled  review  strategies. 

Hannafin  and  Colamaio  (1986)  compared  three  strategies 
of  review  during  a  computer-based  lesson  on  cardiopulmonary 
resuscitation:  designer-imposed  strategy,  learner-selected 

strategy,  and  linear  strategy.  For  each  strategy,  12 
questions  were  presented  during  the  lesson.  In  the  designer- 
imposed  strategy,  if  a  question  was  answered  correctly,  the 
lesson  advanced  to  the  next  segment.  If  the  question  was 
answered  incorrectly,  a  review  of  the  appropriate  video 
segment  was  shown  and  the  question  was  repeated.  If  the 
question  was  answered  incorrectly  on  the  second  attempt,  the 
correct  answer  was  presented  and  the  lesson  advanced  to  the 
next  segment .  In  the  learner-selected  strategy,  the  students 
were  given  options  of  reviewing  each  video  segment,  repeating 
each  question,  or  advancing  to  the  next  segment  regardless  of 
whether  their  answers  were  correct  or  incorrect.  Finally,  in 
the  linear  strategy,  students  observed  a  preset  sequence  of 
video  segments,  and  responses  to  questions  were  followed  only 
by  feedback  about  whether  their  answers  were  correct  or 
incorrect.  No  video  segment  reviews,  question  repetitions, 
or  sequence  choices  were  presented  to  the  students  in  this 
condition.  Subjects  in  the  designer-imposed  and  learner- 
selected  conditions  scored  higher  on  a  posttest  than  did 
subjects  in  the  linear  condition.  However,  there  were  no 
differences  between  scores  for  the  designer-imposed  and 
learner-selected  strategies . 

Finally,  Kinzie  and  Sullivan  (1988)  compared  two 
strategies  for  review  following  incorrect  responses  in  a 
computer-based  science  lesson.  A  learner-control  strategy 
permitted  the  student  to  decide  whether  to  review  material  or 
to  proceed  with  the  lesson.  A  program-control  strategy 
required  each  student  to  review  the  material.  Posttests 
showed  no  differences  between  the  two  strategies. 


Summary 

A  few  of  the  experiments  comparing  methods  of  informa¬ 
tion  presentation  have  produced  results  that  can  help 
designers  of  CBI.  There  is  some  support  for  increasing  the 
stimulus  display  duration  following  correct  responses  so  that 
advanced  students  may  gradually  take  control  of  stimulus 
display  duration  (Tennyson  &  Park,  1984;  Tennyson,  Park,  & 
Christensen,  1985;  Tennyson,  Welsh,  Christensen,  &  Hajovy, 
1985) ,  but  this  contradicts  the  recommendations  made  by 
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Alessi  and  Trollip  (1985),  Eberts  and  Brock  (1984),  and 
Schneider  et  al.  (1982).  Tennyson's  research  also  demon¬ 
strates  an  effective  trial  sequencing  strategy  for  concept 
acquisition  (Park  &  Tennyson,  1980,  1986/  Tennyson,  Park,  & 
Christensen,  1985;  Tennyson,  Welsh,  Christensen,  &  Hajovy, 

1985)  and  provides  weak  support  for  sequencing  lesson  mate¬ 
rial  from  simple  to  complex  (Tennyson,  Welsh,  Christensen,  & 
Hajovy,  1985) .  Wightman  and  Lintern's  (1985)  review  of 
empirical  research  suggests  that  part-task  training  should 
precede  whole-task  training  only  for  tasks  that  can  be 
partitioned  temporally  and  recombined  in  a  backward  chaining 
procedure.  Finally,  one  experiment  demonstrated  that 
animated  graphics  are  more  effective  than  static  graphics  for 
training  in  the  perception  of  dynamic  visual  stimuli 
(McDonald  &  Whitehill,  1987) . 

Many  of  the  published  guidelines  and  recommendations  are 
either  not  supported  or  not  addressed  by  research.  Experi¬ 
ments  comparing  lessons  with  and  without  orienting  instruc¬ 
tions  or  objectives  have  produced  contradictory  results 
(Hannafin  et  al . ,  1986;  Ho  et  al.,  1986;  Rieber  &  Hannafin, 

1986)  .  Furthermore,  the  research  does  not  address  the 
assertion  that  orienting  instructions  and  objectives  limit 
the  amount  or  breadth  of  learning.  Research  support  for 
student  control  of  trial  sequencing  comes  from  experiments 
with  possibly  inadequate  comparison  conditions  (Barsam  & 
Simutis,  1984;  Gray,  1987) .  One  experiment  fails  to  support 
the  recommendation  to  present  information  before  practice 
(Lahey,  1981) .  No  research  was  located  that  addresses 
recommendations  to  present  concrete  material  before  abstract 
material  or  to  use  temporal  versus  random  sequencing  strate¬ 
gies  (cf.,  Barsam  &  Simutis,  1984).  Recommendations  to  avoid 
presenting  material  that  the  student  already  knows  and  to 
avoid  varying  the  difficulty  level  of  material  are  not 
addressed  by  empirical  research.  There  is  also  no  research 
that  addresses  recommendations  about  student  control  of 
difficulty  level,  the  use  of  graphics,  graphics  highlighting, 
placement  of  graphics,  or  quality  of  graphics.  Finally, 
research  shows  no  differences  between  student  control  and 
computer  control  of  reviews. 


Questions 

One  characteristic  of  nearly  all  training  media  is  the 
presentation  of  questions  to  test  the  student's  understanding 
of  the  subject  matter  presented  during  instruction.  Text¬ 
books,  educational  films,  tapes,  and  lecturers  frequently 
query  students  in  an  attempt  to  introduce  lesson  material, 
review  material,  or  test  for  comprehension  of  material. 
Whether  and  when  to  present  questions,  how  to  present 
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questions,  and  how  subjects  should  respond  to  questions  are 
issues  that  must  be  addressed  by  designers  of  CBI . 


Interactivity 

Eberts  and  Brock  (1984)  advise  that  training  systems 
should  be  designed  such  that  students  can  interact  with  the 
training  program.  Hannafin  (1985)  suggests  that  the  effec¬ 
tive  ingredients  of  interactivity  are  question,  response,  and 
feedback  procedures.  Response  and  feedback  procedures  are 
discussed  in  later  sections.  The  importance  of  questions  is 
discussed  in  the  following  paragraphs. 

Guidelines .  Most  authors  of  instructional  design  guide¬ 
lines  agree  that  questions  are  essential  in  CBI.  For 
example,  Jonassen  and  Hannum  (1987)  advise  designers  to  ask 
questions  about  material  before,  during,  or  after  instruc¬ 
tion.  Wager  and  Wager  (1985)  assert  that  postlesson 
questions  increase  retention  of  material. 

Research .  Experiments  designed  to  compare  computer- 
based  presentations  with  questions  to  those  without  questions 
tend  to  support  the  recommendation  to  include  questions  in 
CBI.  There  is  one  exception.  McMullen  (1986)  compared  three 
kinds  of  CBI  programs  that  differed  in  the  amount  of  inter¬ 
activity  and  feedback.  An  information-only  program  presented 
textual  material  without  any  questions.  A  flashcard-type 
drill  program  presented  questions  after  the  text  presenta¬ 
tion.  Subjects  were  required  to  enter  an  answer,  but  no 
feedback  was  provided  for  responses.  An  educational  game 
program  was  identical  to  the  flashcard-type  drill,  except 
feedback  was  provided  for  responses.  Results  showed  no 
differences  in  posttest  scores  among  the  three  programs. 

On  the  other  hand,  Schaffer  and  Hannafin  (1986)  compared 
four  versions  of  a  videotape  lesson  that  varied  in  the  level 
of  interactivity  and  complexity  of  feedback.  In  one  version, 
subjects  viewed  the  videotape  without  questions.  In  a  second 
version,  subjects  viewed  the  videotape  and  were  presented 
with  questions  but  were  provided  no  feedback  for  their 
responses.  In  a  third  version,  subjects  viewed  the  videotape 
and  questions  and  were  informed  whether  they  were  correct  or 
incorrect  in  their  responses.  Finally,  in  a  fourth  version, 
subjects  viewed  the  videotape  and  questions,  were  provided 
with  feedback,  and  when  incorrect,  were  shown  a  review  of  the 
section  of  videotape  containing  the  information.  Overall, 
posttest  scores  increased  as  a  function  of  interactivity  and 
feedback  complexity. 
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Two  experiments  showed  that  presenting  questions  during 
a  lesson  is  more  effective  than  highlighting  the  same 
material.  However,  the  effect  only  holds  for  the  material 
that  was  highlighted  or  questioned.  Schloss,  Schloss,  and 
Cartwright  (1985)  presented  two  versions  of  a  computer-based 
lesson  to  groups  of  subjects.  One  version  asked  open-ended, 
sentence  completion  questions  at  various  intervals  during  the 
lesson.  The  other  version  presented  highlights,  instead  of 
questions,  at  various  intervals  during  the  lesson.  A  high¬ 
light  was  defined  as  a  sentence  entailing  a  question  and  its 
corresponding  answer  from  the  other  version  of  the  lesson. 

All  subjects  were  given  two  posttests  after  viewing  the 
lesson.  One  test  contained  16  multiple-choice  questions 
paralleling  16  of  the  questions  (or  highlights)  from  the 
previous  lesson.  The  other  test  contained  16  questions 
paraphrased  from  the  text  of  the  previous  lesson.  Subjects 
who  viewed  the  lesson  with  questions  performed  better  on  the 
former  posttest  than  did  the  subjects  who  viewed  the  lesson 
with  the  highlights.  There  were  no  differences  between 
groups  on  the  latter  posttest. 

Schloss,  Sindelar,  Cartwright,  and  Schloss  (1986) 
replicated  the  above  study  using  multiple-choice  questions, 
instead  of  open-ended  questions.  Again,  subjects  who  viewed 
the  lesson  with  questions  performed  better  than  subjects  who 
viewed  the  lesson  with  the  highlights  on  a  posttest  contain¬ 
ing  multiple-choice  questions  paralleling  the  questions  (or 
highlights)  from  the  previous  lesson. 


grelesson  Questions 

Prelesson  questions  may  function  in  a  manner  similar  to 
orienting  instructions  or  statements  of  objectives  by 
preparing  the  student  for  the  forthcoming  lesson  material. 
Recommendations  and  research  about  orienting  instructions  and 
objectives  have  already  been  discussed.  However,  some 
authors  of  instructional  design  guidelines  have  made  specific 
recommendations  about  the  use  of  prelesson  questions. 

Guidelines .  Some  authors  recommend  the  use  of  prelesson 
questions;  others  recommend  that  prelesson  questions  not  be 
used  because  they  may  interfere  with  comprehensive  learning. 

MacLachlan  (1986)  asserts  that  prelesson  questions 
enhance  learning.  Jonassen  and  Hannum  (1987)  recommend  that 
lessons  begin  with  a  pretest.  They  advise  instructional 
designers  to  challenge  students  to  answer  questions  presented 
at  the  beginning  of  a  lesson. 
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In  contrast.  Wager  and  Wager  (1985)  believe  that  pre¬ 
lesson  questions  serve  to  focus  attention  on  information 
related  to  answering  the  questions,  but  may  hinder  the 
retention  of  material  that  is  not  introduced  by  the  ques¬ 
tions  .  Hannafin  and  Hughes  (1986)  agree  that  learners  tend 
to  focus  effort  on  information  related  to  prelesson  questions 
to  the  detriment  of  information  not  presented  in  the 
prelesson  questions. 

Research.  Dalton,  Goodrum,  and  Olsen  (1988)  compared 
three  strategies  for  presenting  prelesson  questions  during  a 
computer-based  lesson  on  division  rules.  Subjects  in  one 
group  received  a  pretest  of  20  items  with  feedback  indicating 
whether  each  response  was  correct  or  incorrect.  Subjects  in 
a  second  group  received  the  same  pretest  until  five  items 
were  missed.  After  missing  five  items,  the  pretest  ended  and 
the  lesson  began.  Subjects  in  a  third  group  received  no  pre¬ 
test.  Posttest  scores  were  significantly  higher  for  subjects 
in  the  second  group  than  in  the  first  and  third  groups. 

Dalton  et  al.  speculate  that  the  20-item  pretest  produced  too 
much  failure  prior  to  the  lesson  and  may  have  decreased 
student  motivation.  However,  the  brief  pretest  was  superior 
to  no  pretest,  suggesting  that  limited  prelesson  questioning 
enhances  learning. 


Qtfestion-IyK&s 

Most  computer-based  instructional  programs  use  multiple- 
choice  questions,  typically  for  the  convenience  of  program¬ 
ming  response  processing  and  remediation  routines.  Yet  there 
are  many  other  types  of  questions  that  can  be  presented 
during  CBI . 

Guidelines .  Authors  of  instructional  design  guidelines 
recommend  that  designers  of  CBI  include  the  following  types 
of  questions: 

•  application  questions, 

•  discrimination  questions,  and 

•  rhetorical  questions. 

Jonassen  and  Hannum  (1987)  recommend  questions  that 
require  students  to  think  about  the  application  of  the  lesson 
material.  Alessi  and  Trollip  (1985)  advocate  questions  that 
require  the  student  to  apply  a  rule  or  principle  to  a  new 
situation  rather  than  to  recall  or  recognize  answers.  Wedman 
and  Stefanich  (1984)  assert  that  test  items  should  require 
the  student  to  apply  principles  (or  perform  procedures)  in 
ways  consistent  with  how  principles  will  be  applied  (or 
procedures  performed)  outside  the  learning  situation.  For 
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concept  acquisition,  Wedman  and  Stefanich  recommend  test 
items  that  require  the  student  to  discriminate  between 
examples  and  non-examples  of  each  concept.  Jonassen  and 
Hannum  (1987)  also  recommend  the  use  of  rhetorical  questions. 

Research.  Experiments  evaluating  various  types  of 
questions  have  not  addressed  the  application,  discrimination, 
and  rhetorical  questioning  strategies  recommended  by  Alessi 
and  Trollip  (1985),  Jonassen  and  Hannum  (1987),  and  Wedman 
and  Stefanich  (1984)  .  Dalton  and  Hannafin  (1987)  found  that 
subjects  learned  more  from  a  lesson  emphasizing  application 
of  concepts  than  from  one  that  presented  definitions  of 
concepts  without  describing  their  application.  However,  this 
experiment  did  not  compare  question  types.  Similarly, 
Tennyson  and  his  associates  (Park  &  Tennyson,  1980,  1986; 
Tennyson,  Park,  &  Christensen,  1985;  Tennyson,  Welsh, 
Christensen,  &  Hajovy,  1985)  used  a  discrimination  procedure 
similar  to  the  one  recommended  by  Wedman  and  Stefanich 
(1984)  .  However,  the  focus  of  the  Tennyson  research  was  on 
trial  sequencing  strategies  and  not  the  effectiveness  of 
discrimination  questions  in  concept  acquisition. 

Schloss  et  al.  (1986)  compared  three  ratios  of  higher 
cognitive  questions  to  factual  questions  in  a  computer-based 
lesson.  They  defined  a  higher  cognitive  question  as  one  that 
requires  a  student  to  generate  an  answer  that  is  not  pre¬ 
sented  directly  in  the  previous  text.  A  factual  question 
requires  a  student  to  recognize  or  recall  information  that  is 
presented  directly  in  the  previous  text.  One  version  of  the 
lesson  contained  15  higher  cognitive  questions  and  45  factual 
questions;  a  second  version  contained  30  higher  cognitive 
questions  and  30  factual  questions;  and  a  third  version 
contained  45  higher  cognitive  questions  and  15  factual 
questions.  Posttest  results  revealed  no  differences  among 
the  three  ratios . 

Merrill  (1987)  compared  computer-based  lessons  with 
high-level  questions  to  lessons  with  low-level  questions. 
Lessons  with  high-level  questions  produced  higher  posttest 
scores  than  lessons  with  low-level  questions.  Unfortunately, 
Merrill  provides  no  definition  or  explanation  of  a  high-level 
or  low-level  question.  So,  while  Schloss  et  al.  (1986) 
demonstrated  that  there  is  no  need  to  distinguish  between 
higher  cognitive  and  factual  questions  when  designing  CBI, 
Merrill  demonstrated  a  need  to  distinguish  between  high-level 
and  low-level  questions;  however,  insufficient  explanation  of 
that  distinction  reduces  the  usefulness  of  his  research 
results . 
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In  addition  to  decisions  about  whether  to  ask  questions 
and  the  type  of  questions  to  ask,  designers  must  decide  when 
and  where  to  present  questions  during  a  computer-based 
lesson.  The  issue  of  prelesson  questions  has  been  discussed 
previously;  this  section  discusses  the  placement  of  questions 
within  a  lesson. 

Guidelines .  Authors  of  instructional  design  guidelines 
provide  very  little  guidance  about  the  placement  of  questions 
within  a  lesson.  Alessi  and  Trollip  (1985)  suggest  that 
questions  should  occur  "frequently"  during  a  lesson,  and  that 
only  one  question  should  appear  on  each  display  during  a 
computer-based  test.  Wager  and  Wager  (1985)  recommend  that 
each  question  be  placed  after  the  text  passage  or  diagram  to 
which  it  refers. 

Research.  Very  little  research  has  been  conducted  to 
support  decisions  about  the  placement  of  questions.  Schloss 
et  al.  (1985)  investigated  the  placement  of  questions  in  a 
computer-based  lesson  by  comparing  three  ratios  of  questions 
to  lesson  frames.  One  program  presented  a  question  after 
every  lesson  frame  (1:1);  a  second  program  presented  three 
questions  after  every  three  lesson  frames  (3:3);  a  third 
program  presented  five  questions  after  every  five  frames 
(5:5);  and  a  fourth  program  presented  all  90  frames  before 
presenting  all  90  questions  (90:90).  There  were  no  differ¬ 
ences  in  posttest  scores  due  to  the  different  quest ion/ frame 
ratios.  The  results  of  this  research  do  not  support  the 
recommendation  that  questions  should  occur  frequently  during 
a  lesson  (Alessi  &  Trollip,  1985) . 


Number  of  Questions  or  Problems 

Most  CBI  tutorials,  drills,  and  tests  are  designed  to  be 
interactive  by  presenting  discrete  questions  for  students  to 
answer  or  problems  for  students  to  solve.  Simulations  also 
may  be  divided  into  discrete  segments  for  practice  and 
feedback.  Designers  of  CBI  should  present  as  many  opportuni¬ 
ties  for  interactivity  as  are  necessary  to  achieve  the  train¬ 
ing  objective.  However,  it  is  not  obvious  how  the  number  of 
questions  or  problems  should  be  determined.  The  number  could 
be  fixed  for  all  students  receiving  the  training,  or  the 
number  could  be  based  upon  each  student's  performance  during 
the  lesson.  As  with  other  decisions  during  CBI  development, 
the  student  could  be  given  control  of  the  number  of  questions 
or  programs  presented. 
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Guidelines .  Authors  of  instructional  design  guidelines 
make  contradictory  recommendations  about  the  number  of 
questions  or  problems  required  in  CBI .  Some  authors  claim 
that  the  student  should  control  the  number  of  questions  or 
problems;  others  claim  that  the  computer  should  be  programmed 
to  control  adaptively  the  number  of  questions  or  problems. 

Jonassen  and  Hannum  (1987)  advise  designers  to  let 
students  select  the  number  of  practice  problems  they  want  to 
work  in  a  lesson.  Specifically,  they  recommend  that  (a)  the 
computer  should  ask  students  after  each  practice  problem 
whether  they  want  to  work  another  practice  problem  or  to 
discontinue  the  lesson,  and  (b)  students  should  be  allowed  to 
stop  working  practice  problems  and  return  to  other  parts  of 
the  lesson  whenever  they  choose.  Similarly,  Kearsley  (1986) 
asserts  that  programs  should  allow  students  to  skip 
questions . 

Jonassen  and  Hannum  (1987)  also  advise  designers  to 
provide  more  examples  and  exercises  for  students  with  low 
prior  knowledge  or  low  within-lesson  performance;  further¬ 
more,  they  suggest  that  the  number  should  be  based  upon 
student  achievement  during  the  lesson.  The  computer  could 
either  advise  students  to  solve  an  appropriate  number  of 
problems,  or  it  could  simply  present  the  appropriate  number. 

Research.  Empirical  research  tends  to  support  Jonassen 
and  Hannum 's  latter  recommendation  about  controlling  the 
number  of  problems  in  CBI.  Specifically,  it  appears  most 
effective  to  program  the  computer  to  determine  the  number  of 
problems  necessary  for  each  student  to  master  the  content . 
However,  if  the  computer  simply  informs  the  students  of  the 
number  of  problems  necessary  for  mastery,  students  may 
effectively  select  the  number  of  problems  themselves. 

Tennyson  (1980)  compared  three  strategies  for  selecting 
the  number  of  problems  in  a  concept  acquisition  task.  In  the 
student-control  strategy,  the  student  decided  when  to  stop 
the  practice  problems  and  take  the  posttest.  In  the 
computer-control  strategy,  the  computer  selected  the  number 
of  problems  based  upon  an  assessment  of  the  student's  pretest 
and  lesson  performance.  In  the  third  strategy,  called 
student  control  with  advisement,  the  computer  advised  the 
student  about  the  number  of  problems  necessary  for  mastery 
based  upon  an  assessment  of  the  student's  pretest  and  lesson 
performance,  but  the  student  decided  when  to  stop  the  prob¬ 
lems  and  to  take  the  posttest .  Students  who  were  advised 
about  the  number  of  problems  had  higher  posttest  scores  than 
students  who  controlled  the  number  of  problems  without 
advisement .  There  were  no  differences  in  posttest  scores 
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between  students  who  were  advised  and  students  whose  problems 
were  controlled  by  the  computer.  However,  students  in  the 
advisement  condition  selected  fewer  problems  and  took  less 
time  than  students  in  the  computer-control  condition. 

Tennyson  (1981)  and  Johansen  and  Tennyson  (1983)  repli¬ 
cated  this  experiment  with  different  age  subjects  and  with  a 
rule  learning  task  instead  of  a  concept  acquisition  task. 

The  results  were  reproduced.  From  the  Tennyson  research, 
designers  of  CBI  might  conclude  that  student  control  over  the 
number  of  problems  is  effective  only  if  the  students  are 
advised  about  the  number  of  problems  to  select .  Results 
reported  by  Goetzfried  and  Hannafin  (1985)  appear  to 
contradict  this  conclusion. 

Goetzfried  and  Hannafin  (1985)  compared  three  versions 
of  a  computer-based  lesson  in  mathematics.  In  the  student- 
control  version,  the  students  controlled  the  presentation  of 
the  problems,  but  they  were  advised  to  solve  at  least  four 
problems  before  advancing  in  the  lesson.  In  the  adaptive 
computer-control  version,  the  students  did  not  control  the 
presentation  of  the  problems;  rather,  they  received  addi¬ 
tional  instruction  or  problems  according  to  the  accuracy  of 
their  responses  during  the  lesson.  In  the  linear  computer- 
control  version,  the  computer  presented  a  fixed  sequence  and 
number  of  problems.  In  this  version,  the  students  were  not 
permitted  to  respond;  instead,  the  problems  were  presented 
for  student  viewing,  and  the  correct  solution  was  presented 
on  the  following  frame.  There  were  no  differences  in  post¬ 
test  scores  among  these  three  versions,  and  the  linear 
computer-control  version  took  significantly  less  time  to 
complete . 

While  the  results  of  the  Goetzfried  and  Hannafin 
experiment  seem  contradictory  to  those  of  the  Tennyson 
research,  the  two  procedures  differed  substantially.  First, 
the  algorithms  for  adaptive  problems  number  selection  were 
different.  In  the  Goetzfried  and  Hannafin  (1985)  computer- 
control/adaptive  condition,  "each  student  was  required  to 
solve  correctly  all  four  problems  in  each  section  before 
advancing  to  the  next  divisibility  rule”  (p.  274) .  The 
Tennyson  research  used  a  complex  algorithm  developed  from 
Bayesian  probability  theory  (see  Rothen  &  Tennyson,  1978) . 
Consequently,  the  advisement  procedures  were  different. 
Goeztfried  and  Hannafin  advised  each  subject  to  "solve  at 
least  four  problems  correctly  before  advancing  to  the  next 
section"  (p.  274).  Tennyson's  advice  came  from  the  Bayesian 
algorithm.  Finally,  Goetzfried  and  Hannafin 's  subjects  were 
seventh  grade  students  enrolled  in  remedial  mathematics 
classes;  Tennyson's  subjects  were  average  high  school  and 
college  students. 
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The  student's  response  in  CBI  might  be  an  important 
variable  in  training  effectiveness.  Multiple-choice  ques¬ 
tions,  the  most  common  type,  require  students  to  select  an 
answer  by  typing  a  single  character  or  moving  a  cursor  to  the 
selected  answer.  Fill-in-the-blank.  questions  require  more 
typing.  Simulations  may  require  student  responses  of  an 
entirely  different  nature. 

Guidelines .  Authors  of  instructional  design  guidelines 
offer  the  following,  somewhat  incompatible,  recommendations 
about  answering  questions  during  CBI: 

•  the  student  should  be  permitted  to  see  the  answers  to 
questions  before  responding, 

•  the  student  should  not  be  permitted  to  see  the  answers 
to  questions  before  responding, 

•  overt  responses  are  not  necessary  and  should  not  be 
required, 

•  overt  responses  are  necessary  and  should  be  required, 

•  the  student  should  be  permitted  to  change  an  answer  at 
any  point  during  a  quiz,  and 

•  the  student  should  control  the  sequence  of  questions. 

Alessi  and  Trollip  (1985)  recommend  that  designers  allow 
students  to  see  the  answer  to  a  question  upon  request  during 
computer-based  drills.  However,  Jonassen  and  Hannum  (1987) 
suggest  that  (a)  the  student  should  be  required  to  generate 
an  answer  first  and  then  receive  feedbacJc,  and  (b)  the 
student  should  not  be  permitted  to  look  back  through  the 
presentation  for  answers. 

Jonassen  and  Hannum  (1987)  note  that  students  can 
respond  covertly  as  well  as  overtly.  Asserting  that  overt 
responding  is  not  necessary  for  learning,  they  recommend  that 
designers  give  directions  to  students  to  stimulate  covert 
responding.  However,  they  contradict  themselves  when  they 
state  that  "responses  requiring  multiple  keypresses  are 
required  for  deeper/more  meaningful  mental  processing" 

(p.  11) .  Wager  and  Wager  (1985)  advise  designers  to  ensure 
that  the  student  make  a  substantive  response  before  being 
shown  the  answer. 

Kearsley  (1986)  recommends  that  programs  permit  students 
to  change  their  answers  to  any  question  at  any  point  during  a 
pretest,  posttest,  or  mid- lesson  quiz.  Furthermore,  he 
advocates  that  programs  permit  students  to  back  up  and  skip 
questions  during  pretests,  posttests,  and  exams. 
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Rt^.searr.h .  None  of  the  recommendations  about  student 
responses  are  addressed  by  computer-based  instructional 
strategies  research.  One  experiment  suggests  that  a  brief 
forced  response-delay  will  enhance  scores  on  multiple -choice 
tests.  Stokes,  Halcomb,  and  Slovacek  (1988)  presented 
computer-based  quizzes  to  three  groups  of  students  in  a 
psychology  course.  The  groups  differed  in  the  period  of 
forced  response-delay  following  the  presentation  of  a 
multiple-choice  question:  0-sec,  30-sec,  and  60-sec. 

Subjects  in  the  30-sec  group  had  significantly  higher  quiz 
scores  than  subjects  in  the  other  two  groups.  However,  there 
were  no  differences  among  groups  on  final  examination  scores 
or  course  grades.  The  brief  response-delay  appeared  to 
improve  immediate  recognition  of  the  correct  answers,  but  it 
did  not  affect  retention  over  a  longer  period. 


Summary 

The  recommendation  to  include  questions  in  CBI  is 
generally  supported  by  the  research  literature  (Schaffer  & 
Hannafin,  1986;  Schloss  et  al.,  1985;  Schloss  et  al . ,  1986; 
but  cf.,  McMullen,  1986).  However,  there  is  very  little 
research  to  guide  designers  of  CBI  in  making  decisions  about 
specific  question  and  response  strategies.  There  is  some 
evidence  that  a  brief  prelesson  quiz  will  enhance  posttest 
performance  (Dalton  et  al.,  1988).  Merrill  (1987)  has  shown 
that  high-level  questions  are  superior  to  low-level  ques¬ 
tions,  although  the  distinction  between  the  two  is  unclear. 
Tennyson's  research  suggests  that  students  can  effectively 
control  the  number  of  practice  problems  if  the  computer  is 
programmed  to  (a)  analyze  each  student's  achievement  during  a 
lesson  and  (b)  present  advice  about  the  appropriate  number  of 
problems  required  for  mastery  (Johansen  &  Tennyson,  1983; 
Tennyson,  1980;  Tennyson,  1981) .  Finally,  there  is  some 
evidence  that  a  brief,  forced  delay  of  response  following  the 
presentation  of  a  multiple-choice  question  increases  the 
probability  that  the  student  will  answer  correctly  (Stokes  et 
al.,  1988). 

Most  of  the  guidelines  and  recommendations  about  ques¬ 
tion  and  response  strategies  remain  unsupported  by  empirical 
research.  No  research  was  located  to  evaluate  the  relative 
effectiveness  of  application,  discrimination,  or  rhetorical 
cjuestions.  There  is  no  experimental  support  for  presenting 
questions  "frequently"  during  a  lesson.  No  research  was 
located  to  support  the  recommendations  to  permit  students  to 
see  answers  or  to  prevent  students  from  seeing  answers  before 
responding.  No  experiments  were  found  that  address  the  issue 
of  overt  versus  covert  responding.  Finally,  no  experiments 
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were  found  that  address  recommendations  to  permit  students  to 
change  answers  at  any  point  during  a  quiz  or  to  control  the 
sequence  of  questions. 


Programming  Response  Feedback  and  Remediation  Strategies 

If  interactivity  is  a  critical  element  of  CBI,  decisions 
about  feedback  and  remediation  are  as  important  as  decisions 
about  the  presentation  of  material  and  questions.  Designers 
must  determine  what  sorts  of  events  should  follow  correct  and 
incorrect  responses,  when  these  events  should  occur,  and  how 
these  events  should  be  presented  on  the  screen.  This  section 
presents  some  of  the  guidelines  and  empirical  research  per¬ 
taining  to  feedback  for  correct  and  incorrect  responses, 
remediation  strategies,  latency  of  feedback,  and  placement  of 
feedback . 


Feedback  for  Correct  Responses 

Instructional  designers  must  decide  what  sorts  of 
events,  if  any,  should  follow  correct  responses  by  students. 
The  designer  must  decide  if  and  how  the  computer  should 
reinforce  correct  responding. 

Guidelines .  Authors  of  instructional  design  guidelines 
make  the  following  three  recommendations  about  feedback  for 
correct  responses : 

•  feedback  is  unnecessary  and  should  not  be  used, 

•  feedback  should  be  brief,  and 

'  •  feedback  should  explain  why  the  responses  are  correct . 

Cohen  (1985)  asserts  that  feedback  after  a  correct 
response  is  not  as  important  as  feedback  after  an  incorrect 
response  in  CBI.  However,  Wager  and  Wager  (1985)  recommend  a 
short  affirmation  of  each  correct  response.  They  warn 
designers  to  avoid  correct  response  feedback  that  is  time- 
consuming.  Conversely,  Jonassen  and  Hannum  (1987)  warn 
designers  to  avoid  feedback  that  simply  indicates  whether  an 
answer  is  correct  or  incorrect.  Instead,  they  recommend  that 
correct  answers  be  followed  by  feedback  indicating  that  the 
answer  is  correct  and  explaining  why  it  is  correct .  Wager 
and  Wager  (1985)  suggest  that  the  computer  reinforce  correct 
responses  by  displaying  the  answer  in  the  context  of  the  item 
whenever  possible. 

Research.  No  research  was  located  that  addresses  recom¬ 
mendations  about  feedback  for  correct  responses  in  CBI. 
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As  with  feedback  for  correct  responses,  designers  of  CBI 
must  determine  the  most  effective  consequences  of  incorrect 
responses  during  a  lesson.  In  addition  to  informing  students 
that  a  response  is  incorrect,  the  computer  is  capable  of 
providing  remedial  training  or  instruction.  Therefore,  it  is 
important  to  determine  the  most  effective  methods  for 
remediating  errors  in  CBI. 

Guidelines .  Authors  of  instructional  design  guidelines 
provide  the  following  recommendations  for  feedback  and 
remediation  of  errors  in  CBI; 

•  corrective  feedback  is  necessary  and  should  be 
provided, 

•  feedback  should  be  specific  to  the  type  of  error, 

•  novel,  entertaining,  or  auditory  stimuli  should  not  be 
used  as  feedback  for  incorrect  responses,  and 

•  multiple  trials  should  be  presented  when  an  item  is 
missed. 

In  addition,  some  authors  make  specific  recommendations  for 
remediation  in  cases  of  multiple  incorrect  responses  and 
partially  incorrect  responses. 

As  stated  previously,  Jonassen  and  Hannum  (1987)  warn 
designers  to  avoid  presenting  feedback  that  simply  indicates 
whether  an  answer  is  correct  or  incorrect.  Feedback  for 
incorrect  responses  should  be  corrective  (Alessi  &  Trollip, 
1985;  Kearsley,  1986) .  Specifically,  Wager  and  Wager  (1985) 
advise  that  feedback  should  focus  on  correcting  misconcep¬ 
tions  represented  by  incorrect  answers.  Cohen  (1985)  agrees 
with  this  advice,  noting  that  informational  feedback  is 
better  than  simple  knowledge  of  results  following  incorrect 
responses . 

Different  types  of  errors  should  produce  different  types 
of  feedback  (Jonassen  &  Hannum,  1987)  .  For  example,  Alessi 
and  Trollip  (1985)  suggest  a  special  method  of  feedback 
following  errors  in  discrimination  problems.  Jonassen  and 
Hannum  (1987)  suggest  that  if  the  specific  error  can  be 
determined,  feedback  should  indicate  what  error  was  made,  why 
it  was  an  error,  and  how  it  can  be  corrected.  If  the 
specific  error  cannot  be  determined,  the  computer  should 
present  the  correct  answer  and  an  explanation  of  why  it  is 
correct . 


Wager  and  Wager  (1985)  warn  designers  to  avoid  correc¬ 
tive  feedback  that  is  entertaining  or  novel.  Jonassen  and 


Hannum  (1987)  add  a  warning  never  to  use  auditory  cues  to 
signal  incorrect  responses. 

Alessi  and  Trollip  (1985)  recommend  that  questions 
answered  incorrectly  be  repeated  at  variable  intervals  later 
in  the  instructional  session.  Jonassen  and  Hannum  (1987) 
reinforce  this  recommendation  by  suggesting  that  the  computer 
select  items  for  review.  If  a  program  permits  multiple 
opportunities  to  attempt  a  correct  answer,  Wager  and  Wager 
(1985)  recommend  providing  increasingly  informative  feedback 
after  each  successive  wrong  answer.  However,  they  caution 
designers  to  restrict  the  number  of  trials  allowed  before 
presenting  the  student  with  the  correct  answer  or  corrective 
feedback.  If  an  answer  is  partially  incorrect  (and  partially 
correct) ,  feedback  should  prompt  a  correct  response  by 
telling  students  why  their  answers  are  incomplete  (Wager  & 
Wager,  1985) . 

Research .  Several  experiments  have  been  conducted  to 
examine  the  effects  of  different  types  of  feedback  and 
remediation  strategies  for  incorrect  responses .  Consequent¬ 
ly,  there  is  some  experimental  support  for  the  recommendation 
to  provide  corrective  feedback  rather  than  only  knowledge  of 
results.  Two  of  the  conditions  in  the  Schaffer  and  Hannafin 
(1986)  experiment,  discussed  previously,  provide  a  test  of 
error  feedback  complexity.  Subjects  in  both  conditions 
viewed  a  computer-based  videotape  lesson  and  answered  ques¬ 
tions.  Subjects  in  one  condition  were  only  informed  whether 
they  were  correct  or  incorrect  in  their  responses.  Subjects 
in  the  other  condition  were  provided  with  similar  feedback 
and,  when  answers  were  incorrect,  were  shown  a  review  of  the 
section  of  videotape  containing  information  relevant  to  the 
question.  Posttest  scores  were  higher  for  subjects  who  were 
shown  the  reviews . 

Similarly,  Waldrop,  Justen,  and  Thomas  (1986)  compared 
three  types  of  feedback  for  responses  on  a  20-trial  computer- 
based  drill.  In  the  minimal  feedback  condition,  subjects 
were  merely  informed  whether  their  responses  were  correct  or 
incorrect.  In  the  extended  feedback  condition,  subjects  were 
not  only  informed  of  the  correctness  of  the  answers,  but  were 
also  provided  with  an  explanation  of  the  correct  answer.  In 
a  third  condition,  minimal  feedback  was  provided  for  the 
first  two  incorrect  responses  on  a  problem,  but  extended 
feedback  was  provided  if  the  subject  answered  incorrectly  a 
third  time.  Subjects  in  the  extended  feedback  condition 
scored  higher  on  a  posttest  than  subjects  in  the  minimal 
feedback  condition.  However,  posttest  scores  of  subjects  in 
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the  minimal  plus  extended  feedback  condition  were  no  differ¬ 
ent  than  the  scores  of  the  subjects  in  the  other  two 
conditions . 

Two  experiments  examined  the  complexity  of  remediation 
strategies  for  incorrect  responses ,  Seigel  and  Misselt 
(1984)  compared  three  strategies  of  feedback  following 
incorrect  responses  in  a  flashcard-type  drill  for  learning 
foreign  words.  In  the  first  strategy,  all  incorrect 
responses  were  followed  by  a  presentation  of  the  correct 
answer.  In  the  second  strategy,  errors  were  classified  as 
out-of-list  errors  or  discrimination  errors.  An  out-of-list 
error  occurred  when  a  subject's  incorrect  answer  (a  transla¬ 
tion)  was  a  word  that  was  not  being  taught  in  the  lesson.  A 
discrimination  error  occurred  when  a  subject's  incorrect 
answer  was  a  word  that  was  taught  in  the  lesson  but  was 
incorrect  on  that  particular  trial.  Out-of-list  errors  were 
followed  by  a  presentation  of  the  correct  answer  (as  in  the 
first  strategy) .  Discrimination  errors  were  followed  by  a 
presentation  of  the  correct  answer  and  the  correct  transla¬ 
tion  for  the  subject's  incorrect  response.  The  third  strat¬ 
egy  was  identical  to  the  second,  except  that  discrimination 
errors  were  also  followed  by  additional  discrimination  train¬ 
ing.  Posttest  scores  showed  no  differences  between  the  first 
and  second  remediation  strategies.  However,  subjects  using 
the  third  strategy  had  fewer  posttest  errors  than  the  other 
subjects . 

Merrill  (1987)  compared  two  types  of  feedback  for  errors 
during  a  computer-based  science  lesson.  Corrective  feedback 
was  a  complete  description  of  the  correct  answer.  Attribute 
isolation  feedback  was  not  fully  explained  in  the  report. 
However,  the  attribute  isolation  feedback  may  have  been  less 
comprehensive,  more  specific,  and  more  detailed  than  the 
corrective  feedback.  According  to  Merrill,  "attribute  isola¬ 
tion  helps  to  focus  attention  on  the  critical  and  variable 
attributes  of  a  concept"  (p.  18) .  Results  showed  no  differ¬ 
ences  in  posttest  scores  between  these  two  types  of  feedback. 

Seigel  and  Misselt  (1984)  demonstrated  that  multiple 
repetitions  of  a  problem  following  an  incorrect  response 
produced  higher  posttest  scores  than  a  single  immediate 
repetition  of  the  problem.  They  compared  two  strategies  for 
repeating  problems  to  which  incorrect  responses  occur.  In 
one  strategy,  the  problem  is  repeated  immediately  following 
the  incorrect  response.  In  the  other  strategy,  the  problem 
is  repeated  on  the  next,  4th,  and  9th  trials  following  the 
error.  Subjects  exposed  to  the  multiple  repetition  strategy 
made  significantly  fewer  posttest  errors  than  subjects 
exposed  to  the  single  repetition  strategy.  These  results 
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support  the  recommendation  to  provide  multiple  opportunities 
to  respond  to  a  missed  item. 

The  Waldrop  et  al .  (1986)  experiment,  described 
previously,  serves  as  a  weak  test  of  the  recommendation  that 
multiple  errors  be  followed  by  increasingly  more  informative 
feedback  (Wager  &  Wager,  1985) .  In  this  case,  the  results 
fail  to  support  the  recommendation;  that  is,  the  posttest 
scores  for  subjects  in  the  minimal  plus  extended  feedback 
condition  were  no  different  than  the  scores  for  the  other  two 
conditions.  Thorkildsen  and  Friedman  (1986)  examined  a 
different  approach  to  remediating  multiple  incorrect 
responses.  They  compared  two  branching  strategies  termed 
extensive  (EXT)  and  minimal  (MIN) .  For  both  strategies,  the 
first  occurrence  of  an  incorrect  answer  to  a  particular 
question  is  followed  by  a  repetition  of  the  question.  After 
a  second  incorrect  answer  to  the  question,  EXT  branches  to  a 
simpler  question  than  the  original  one,  while  MIN  presents 
the  original  question  for  the  third  time.  After  a  third 
incorrect  response  to  a  question,  EXT  presents  an  even 
simpler  question  with  a  prompt  for  the  correct  answer,  while 
MIN  presents  the  correct  answer.  Posttest  scores  showed  no 
significant  differences  between  these  two  branching  strate¬ 
gies.  However,  subjects  in  the  EXT  condition  spent  signifi¬ 
cantly  less  time  on  the  system.  Recommendations  about 
effective  remediation  of  multiple  errors  will  probably  have 
to  wait  for  definitive  research  on  effective  remediation  of 
single  errors. 

No  research  was  located  that  addresses  the  recommenda¬ 
tions  to  avoid  novel,  entertaining,  or  auditory  stimuli  in 
error  feedback  (Jonassen  &  Hannum,  1987;  Wager  &  Wager, 

1985) .  Similarly,  no  research  was  located  that  addresses  the 
recommendation  to  provide  corrective  feedback  for  partially 
incorrect  responses  (Wager  &  Wager,  1985) .  However,  if  a 
partially  incorrect  response  is  considered  an  incorrect 
response,  research  supports  the  recommendation  to  present 
corrective  feedback  (Schaffer  &  Hannafin,  1986;  Waldrop  et 
al.,  1986)  . 


Latency  of  Feedback 

In  addition  to  decisions  about  the  kind  of  feedback  for 
responses  in  CBI,  designers  must  determine  the  temporal 
parameters  of  feedback.  For  example,  designers  must  decide 
whether  feedback  should  occur  immediately  after  responses  or 
after  some  delay.  If  delayed,  designers  must  determine  how 
much  time  should  pass  between  the  response  and  the  feedback. 
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Guidelines .  Authors  of  instructional  design  guidelines 
make  the  following  contradictory  recommendations  about 
latency  of  feedback: 

•  immediate  feedback  should  always  be  presented, 

•  delayed  feedback  should  always  be  presented,  and 

•  immediate  feedback  should  be  presented  at  some  times 
and  delayed  feedback  should  be  presented  at  other 
times . 

Alessi  and  Trollip  (1985)  recommend  giving  immediate 
feedback  for  incorrect  answers  during  drills.  For  tests, 
detailed  feedback  should  be  given  immediately  after  the  test. 
Cohen  (1985)  agrees  with  this  recommendation,  stating  that 
immediate  feedback  is  better  than  delayed  or  end-of-session 
feedback  for  students  exhibiting  low  mastery  of  the  material. 
Jonassen  and  Hannum  (1987)  also  agree  that  immediate  feedback 
is  more  effective  for  initial  acquisition  of  material.  Addi¬ 
tionally,  they  assert  that  immediate  feedback  about  conse¬ 
quences  of  decisions  is  more  effective  than  delayed  feedback. 
Wheaton  et  al.  (1976)  recommend  that  "information  about  the 
correctness  of  action  should  be  available  quickly”  (p.  78), 
but  that  "delay  of  [feedback]  has  little  or  no  effect  on 
acquisition"  (p .  77). 

On  the  other  hand,  Cohen  (1985)  maintains  that  immediate 
feedback  can  impede  the  pace  of  learning,  and  MacLachlan 
(1986)  asserts  that  students  with  higher  skill  levels  may 
learn  better  under  conditions  of  delayed  feedback.  Cohen 
recommends  immediate  feedback  for  the  initial  acquisition  of 
material  and  for  recognition  or  immediate  recall  of  ideas. 
However,  if  students  have  prior  knowledge  of  material, 
informational  feedback  should  be  delayed,  and  knowledge  of 
results  should  be  presented  immediately  after  each  response. 
End-of-session  feedback  should  be  provided  when  comprehen¬ 
sion,  long-term  retention,  and  application  of  information  are 
the  training  objectives.  However,  Cohen  cautions  that 
delayed  feedback  should  be  presented  no  longer  than  15  or  20 
minutes  after  the  responses  occur.  After  long  periods  with¬ 
out  feedback,  "the  effect  of  the  feedback  becomes  negligible 
and  confusing"  (Cohen,  1985,  p.  36) . 

Jonassen  and  Hannum  (1987)  assert  that  end-of-session 
feedback  facilitates  learning  of  more  abstract  material, 
particularly  for  higher  achievement  learners.  For  computer- 
based  simulations,  Alessi  and  Trollip  (1985)  recommend 
immediate  feedback  (regardless  of  fidelity)  with  beginning 
students,  and  natural  feedback  (regardless  of  immediacy)  with 
more  advanced  students. 


Research .  No  research  was  found  that  supports  the 
recommendations  about  latency  of  feedback  in  CBI .  Gaynor 
(1981)  presented  computer-based  math  problems  to  four  groups 
of  subjects.  One  group  received  immediate  feedback  for 
responses.  A  second  group  received  feedback  after  a  30- 
second  delay.  A  third  group  received  feedback  only  at  the 
end  of  the  session.  A  fourth  group  received  no  feedback  for 
responses.  No  significant  differences  in  posttest  scores 
were  observed  among  the  groups. 

One  experiment  appears  to  support  a  recommendation  to 
delay  corrective  feedback  during  computer-based  simulations. 
Munro,  Fehling,  and  Towne  (1985)  compared  two  strategies  for 
presenting  error  feedback  and  remediation  during  a  computer- 
simulated  air  intercept  controller  task.  For  both  strate¬ 
gies,  errors  were  immediately  followed  by  a  tone  and  a  brief 
message  in  an  area  of  the  computer  display  reserved  for 
instructional  messages.  In  one  strategy,  an  additional 
instructional  message  was  shown  immediately  following  the 
error  notice.  In  the  other  strategy,  the  instructional 
messages  were  not  shown  unless  the  subject  requested  them. 
Subjects  using  the  latter  strategy  committed  fewer  errors 
overall  and  on  the  last  ten  trials  than  the  subjects  using 
the  former  strategy. 

These  results  could  be  taken  as  support  for  a  recommen¬ 
dation  to  delay  corrective  feedback  during  a  simulation,  or 
at  least  to  present  corrective  feedback  under  the  student's 
control.  However,  because  of  the  dynamic  nature  of  the  air 
intercept  controller's  task,  it  is  likely  that  subjects  in 
the  immediate  feedback  condition  observed  fewer  of  the 
instructional  messages  than  subjects  in  the  delayed  feedback 
condition.  The  corrective  feedback  appeared  in  an  area  of 
the  screen  reserved  for  messages,  not  in  the  area  of  the 
screen  that  the  subject  had  to  view  to  perform  the  task.  The 
comparison  in  this  experiment  may  have  been  between  an 
instructional  message  condition  and  a  no-message  condition, 
and  subject  control  and  delay  of  feedback  were  not  the  causal 
factors . 


Placement  of  Feedback 

In  addition  to  the  content  and  latency  of  feedback, 
instructional  designers  must  decide  how  to  structure  the 
feedback  on  the  computer  display.  Wager  and  Wager  (1985) 
recommend  that  feedback  be  placed  on  the  same  screen  as  the 
question  and  below  the  student's  response.  The  results  of 
the  Munro  et  al.  (1985)  experiment  suggest  that  placement  of 
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feedback  is  an  important  issue  in  the  design  of  CBI .  How¬ 
ever,  no  research  has  been  located  to  evaluate  different 
strategies  for  placement  of  feedback. 


Summary 

As  with  the  presentation  of  instructional  material  and 
questions,  there  is  very  little  research  to  guide  instruc¬ 
tional  designers  in  making  decisions  about  feedback  and 
remediation  strategies.  There  is  some  experimental  support 
for  the  recommendation  to  provide  corrective  feedback, 
instead  of  simple  knowledge  of  results,  following  incorrect 
responses  (Schaffer  &  Hannafin,  1986;  Seigel  &  Misselt,  1984; 
Waldrop  et  al . ,  1986).  Also,  there  is  evidence  that  multiple 
repetitions  of  a  problem  following  an  incorrect  response  are 
more  effective  than  a  single  repetition  of  the  problem 
(Seigel  &  Misselt,  1984)  . 

However,  the  research  has  not  unequivocally  determined 
whether  and  how  different  types  of  errors  should  be  followed 
by  different  types  of  feedback  (Merrill,  1987;  Seigel  & 
Misselt,  1984)  .  No  research  was  located  to  show  that  feed¬ 
back  for  correct  responses  is  unnecessary,  should  be  brief, 
or  should  explain  why  a  response  is  correct.  No  evidence  was 
found  that  multiple  errors  should  be  followed  by  increasingly 
informative  feedback  (Thorkildsen  &  Friedman,  1986;  Waldrop 
et  al . ,  1986).  No  research  was  located  that  addresses  recom¬ 
mendations  to  avoid  novel,  entertaining,  or  auditory  stimuli 
in  error  feedback.  No  research  was  located  to  support  any  of 
the  recommendations  about  latency  of  feedback  in  CBI  or  to 
evaluate  different  strategies  for  placement  of  feedback. 


Other  Guidelines  about  Instructional  Strategies 

Authors  of  instructional  design  guidelines  give  far  more 
advice  than  the  empirical  research  currently  supports.  For 
example,  Kearsley  and  Frost  (1985)  warn  designers  to  avoid 
cluttering  the  screen  with  too  much  information.  Certainly, 
a  program  should  not  present  subjects  with  ineffective  visual 
displays,  but  how  much  information  is  too  much?  When  is  a 
screen  cluttered?  Braden  (1986)  and  Kearsley  and  Frost 
suggest  presenting  only  one  primary  concept  or  idea  per 
visual  display.  This  recommendation  lacks  empirical  research 
support.  Furthermore,  Braden  recommends  that  lists  be 
restricted  to  seven  or  fewer  items  per  display. 

Several  authors  offer  guidelines  about  the  presentation 
of  textual  material.  Alessi  and  Trollip  (1985)  recommend 
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normal  upper  and  lower  case  text.  Buehner  (1987)  asserts 
that  double-spaced  text  is  easier  to  read  than  single-  or 
triple-spaced  text.  Wager  and  Wager  (1985)  warn  designers  to 
avoid  abbreviations.  They  advise  designers  to  spell  words 
completely  and  use  complete  sentences.  MacLachlan  (1986) 
opposes  the  use  of  cliches  and  recommends  the  use  of  subordi¬ 
nate  words  and  concepts.  For  example,  MacLachlan  prefers  the 
use  of  a  specific  term,  such  as  "raven, "  over  the  use  of  a 
more  general  term,  such  as  "bird."  No  experiments  were  found 
that  address  such  issues. 

A  few  authors  advocate  mnemonics  in  CBI .  MacLachlan 
(1986)  specifically  recommends  the  use  of  rhyme,  rhythm,  and 
the  method  of  loci  in  textual  material.  Jonassen  and  Hannum 
(1987)  suggest  that  designers  periodically  include  directions 
for  the  student  to  "generate  mental  images  of  the  content" 

(p.  10) .  The  use  of  mnemonics  in  learning  has  empirical 
support  in  traditional  educational  research.  There  is  no 
reason  to  assume  that  the  value  of  mnemonics  would  be  less 
for  CBI. 

Several  recommendations  about  instructional  strategies 
are  so  vague  that  their  value  to  designers  is  questionable. 
For  example,  Eberts  and  Brock  (1984)  recommend  that  CBI 
include  "examples  and  analogies  to  make  the  training  effec¬ 
tive"  (p.  280) .  They  also  advise  designers  to  make  the 
learning  intrinsically  motivating  by  utilizing  challenge, 
fantasy,  and  curiosity  in  the  learning  environment.  Finally, 
they  recommend  that  the  computer  display  should  provide 
information  in  a  manner  that  can  be  used  to  form  an  accurate 
internal  representation  of  the  system  or  concept  being 
trained.  Kearsley  and  Frost  (1985)  advise  designers  to 
"organize  information  functionally  on  the  screen  as  much  as 
possible  to  reduce  confusion  and  unnecessary  cognitive 
processing"  (p.  10).  Finally,  Jonassen  and  Hannum  (1987) 
suggest  that  designers  "organize  material  to  an  appropriate 
top-level  structure;  description,  compare/contrast, 
temporal,  explanation,  definition,  example,  problem  solution, 
causation"  (p.  10) . 


Discussion 

Although  there  are  some  consistencies  in  the  literature, 
the  guidelines  and  research  in  computer-based  instructional 
strategies  are  characterized  by  contradictions .  In  some 
cases,  authors  of  instructional  design  guidelines  contradict 
each  other  with  their  recommendations.  In  other  cases, 
empirical  research  contradicts  the  experts'  recommendations. 
Many  of  the  recommendations,  however,  lack  empirical  research 
altogether . 
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Each  guideline  discussed  in  this  review  was  classified 
by  (a)  whether  authors  of  instructional  guidelines  made 
another  recommendation  that  contradicted  the  guideline  and 
(b)  whether  empirical  research  supports  the  guideline, 
contradicts  the  guideline,  or  does  not  exist.  The  guidelines 
and  their  categories  are  listed  in  Table  1.  If  any  guideline 
contradicted  another  guideline,  both  of  the  guidelines  were 
classified  under  "Some  Authors  Disagree.”  Otherwise,  the 
guidelines  were  classified  under  "No  Disagreement  Among 
Authors."  If  empirical  research  supports  or  contradicts  a 
guideline,  the  guideline  was  classified  under  "Research 


Table  1 

Listing  and  Classification  of  CBI  Guidelines 


Guideline 


No 

Disagreement  Some  Insufffi- 

Among  Authors  Research  Research  cient 

Authors  Disagree  Supports  Contradicts  Research 


1. 

Present  cjuestions 

X 

X 

2. 

Corrective  feedback 

is  necessary  for 
incorrect  responses 

X 

X 

3. 

Multiple  trials 
should  be  presented 
when  an  item  is  missed 

X 

X 

4. 

Present  prelesson 
questions 

X 

X 

5. 

Program  the  con^uter 
to  control  adaptively 
the  number  of  trials 

X 

X 

6. 

Use  graphics  often 

X 

X 

7. 

Provide  increasingly 
informative  feedback 
for  successive  errors 

X 

X 

8. 

Train  under  mild  speed 
stress 

X 

X 

9. 

Present  information 
before  practice 

X 

X 

10. 

Randomize  the  sequence 
of  material 

X 

X 

11. 

Part-task  training 
should  precede  whole- 
task  training 

X 

X 

12. 

Permit  student  to 
control  the  presenta¬ 
tion  of  reviews 

X 

X 

X 


X 
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Listing  and  Classification  of  CBI  Guidelines  (Continued) 


Guideline 


No 

Disagreement  Some  Insufffl- 

Among  Authors  Research  Research  cient 

Authors  Disagree  Supports  Contradicts  Research 


13. 

14. 

15. 


16. 

17. 

18. 

19. 

20. 
21. 
22. 

23. 

24. 

25. 

26. 


27. 


Program  the  computer 
to  control  the 
presentation  of  reviews 


Use  graphics  to 
highlight  important 
material 

Place  graphic 
information  on  the 
seune  screen  as  and  to 
the  left  of  the  text 

High  quality  graphics 
are  unnecessary 

Use  animation 

Present  application 
questions 

Present  discrimina¬ 
tion  qpjestions 

Present  rhetorical 
questions 

Present  questions 
frequently 

Present  each 
question  after  the 
text  passage  to 
which  it  refers 

Permit  student  to 
change  answers  at  any 
point  during  a  quiz 

Permit  student  to 
control  the  sequence 
of  questions 

Feedback  should  be 
specific  to  the  type 
of  error 

Do  not  present  novel, 
entertaining,  or 
auditory  feedback 
for  errors 

Place  feedback  on 
the  same  screen  as 
the  question 


X 


X 


X 

X 

X 

X 

X 

X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 


X 

X 

X 

X 

X 

X 


X 


X 


X 


X 


X 


X 
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Listing  and  Classification  of  CBI  Guidelines  'Continued) 


Guideline 


No 

Disagreement  Some  Insufffi- 

Among  Authors  Research  Research  cient 

Authors  Disagree  Supports  Contradicts  Research 


28.  Orienting  instructions 
are  ineffective 

29.  Present  orienting 
instructions 

30 .  Lesson  objectives  are 
ineffective 

31.  Lesson  objectives 
limit  learning 

32 .  Present  lesson 
objectives 

33.  Present  lesson 
objectives  as  a 
student-controlled 
option 

34.  Permit  student  to 
control  duration  of 
questions  and  graphics 

35.  Shorten  tasks  without 
losing  ijnqjortant 
aspects  for  training 

36.  Always  permit  student 
to  control  the  sequence 
of  instruction 

37 .  Never  permit  student 

to  control  the  sequence 
of  instruction 

38.  Conditionally  permit 
student  to  control  the 
sequence  of  instruction 

39.  Present  concrete 
material  before 
abstract  material 

40.  Use  chronological 
ordering  of  material 

41.  Present  simple 
material  before 
complex  material 

42 .  Do  not  present 
material  that  the 
student  already  knows 

43.  Do  not  vary  the 
difficulty  of  material 
during  the  session 


X 

X 

X 

X 

X 


X 


X 


X 


X 


X 


X 


X 

X 

X 


X 


X 


X 

X 

X 

X 

X 


X 


X 


X 


X 


X 


X 


X 

X 

X 


X 


X 
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Table  1 

Listing  and  Classification  of  CBI  Guidelines  (Continued) 


No 

Disagreement 

Some 

Insuf f f i- 

Among 

Authors 

Research 

Research  dent 

Guideline  Authors 

Disagree 

Supports 

Contradicts  Research 

44. 

Permit  the  student  to 
control  the  difficulty 
level  of  material 

X 

X 

45. 

Do  not  permit  the 
student  to  control  the 

difficulty  level  of 
material 

46. 

Prelesson  questions 

X 

y 

limit  learning 

47  . 

Permit  student  to 
control  the  number 
of  problems 

X 

X 

48. 

Permit  student  to 
see  answer  before 
responding 

X 

X 

49. 

Require  student  to 
respond  before 
presenting  answer 

X 

X 

50. 

Present  questions  that 

V 

Y 

evoke  covert  responses 

51. 

Require  overt  responses 

X 

X 

52. 

Feedback  for  correct 
responses  is 
unnecessary 

X 

X 

53. 

Feedback  for  correct 

responses  should  be 
brief 

X 

X 

54. 

Feedback  for  correct 

responses  should 
explain  why  responses 

X 

X 

are  correct 

55. 

Immediate  feedback 
should  always  be 
presented 

X 

X 

56. 

Delayed  feedback 
should  always  be 
presented 

X 

X 

57. 

Present  immediate 
feedback  sometimes  and 

delayed  feedback  at 
other  times 
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Supports"  or  "Research  Contradicts,"  respectively.  Often,  a 
single  experiment  exists  to  support  or  contradict  a  guide¬ 
line.  Such  evidence  may  or  may  not  be  considered  conclusive, 
depending  upon  the  strength  of  the  research  and  the  speci¬ 
ficity  of  the  guideline.  Finally,  a  guideline  was  classified 
under  "Insufficient  Research"  if  (a)  empirical  research 
produced  mixed  results  and  further  research  is  required,  (b) 
empirical  research  is  inconclusive  because  of  inadequate 
experimental  designs,  or  (c)  empirical  research  on  that 
guideline  was  not  located. 

In  the  following  sections,  each  guideline  is  evaluated 
in  terms  of  its  usefulness  for  computer-based  instructional 
design.  Some  of  the  guidelines  are  self-evident  and  do  not 
require  empirical  validation.  Other  guidelines  are  stated  in 
terms  that  are  so  general  that  it  is  not  possible  to  evaluate 
their  usefulness.  Some  of  the  guidelines  are  very  specific, 
and  an  evaluation  of  their  generalizability  is  required. 
Generalizability  may  exist  on  different  dimensions,  and  CBI 
researchers  and  designers  need  to  understand  the  important 
dimensions  in  educational  technology.  The  present  review  and 
interpretation  of  the  literature  suggests  that  there  are  at 
least  three  important  dimensions  along  which  generalizability 
may  vary:  (a)  training  format  (e.g.,  tutorials,  drills,  or 
simulations),  (b)  training  objectives  (e.g.,  acquisition  vs 
sustainment  training,  verbal/conceptual  vs  nonverbal/ 
procedural  training),  and  (c)  target  population  (e.g.,  high 
school  or  college  students,  military  personnel,  industrial 
workers) . 


Guidelines  Supported  by  Research 

Guidelines  1-5  in  Table  1  are  supported  by  empirical 
research.  Authors  of  guidelines  agree  that  CBI  should  pre¬ 
sent  questions,  corrective  feedback  for  incorrect  responses, 
and  multiple  trials  for  items  that  are  answered  incorrectly. 
The  recommendation  to  present  questions  applies  principally 
to  tutorials.  However,  questions  might  also  be  incorporated 
into  some  types  of  training  simulations,  such  as  those 
designed  to  train  complex  decision-making  skills. 

The  recommendation  to  present  corrective  feedback  means 
that  feedback  that  simply  informs  the  student  of  whether  an 
answer  is  correct  or  incorrect  is  not  as  effective  as  feed¬ 
back  that  presents  more  information.  Precisely  how  much 
information  is  sufficient  probably  depends  upon  the  instruc¬ 
tional  content,  the  purpose  of  the  instruction,  and  the  type 
of  student.  The  recommendation  to  present  multiple  problems 
following  incorrect  responses  is  supported  by  a  single 
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experiment.  The  generalizability  of  this  guideline  needs  to 
be  evaluated. 


Although  some  authors  do  not  agree  that  CBI  should 
present  prelesson  questions  or  that  computers  should  be  pro¬ 
grammed  to  control  adaptively  the  number  of  trials,  empirical 
research  suggests  that  these  strategies  are  effective.  The 
recommendation  to  present  prelesson  questions  is  supported  by 
a  single  experiment.  Therefore,  the  generalizability  of  this 
guideline  should  be  evaluated.  The  recommendation  to  program 
the  computer  to  control  adaptively  the  number  of  trials  is 
supported  by  a  series  of  experiments  in  which  age  of  student 
and  instructional  content  have  varied. 


Guidelines  Contradicted  by  Research 

Although  authors  agree  that  designers  of  CBI  should  use 
graphics  often  and  provide  increasingly  informative  feedback 
for  successive  errors  (guidelines  6  and  7,  Table  1) , 
empirical  research  fails  to  support  these  guidelines .  How¬ 
ever,  the  experimental  evidence  is  not  strong  enough  to 
justify  complete  rejection  of  these  guidelines.  For  example, 
while  experiments  that  compare  graphics  to  text  have  not 
produced  results  suggesting  that  graphics  should  be  used 
often,  no  research  has  been  located  to  show  that  graphics 
should  be  avoided.  Whether  graphics  are  useful  in  CBI 
probably  depends  upon  many  other  factors,  such  as  the 
training  format,  training  objectives,  and  target  population. 

The  recommendation  to  provide  increasingly  informative 
feedback  for  successive  errors  was  contradicted  by  several 
experiments.  Consequently,  its  rejection  may  be  warranted. 
However,  it  seems  reasonable  to  suppose  that  errors  made 
repeatedly  require  more  extensive  remedial  effort  than  errors 
made  only  once  or  twice . 

Some  authors  do  not  agree  with  the  next  six  guidelines 
in  Table  1  (guidelines  8-13),  all  of  which  are  contradicted 
by  empirical  research.  Again,  the  experimental  evidence  does 
not  necessarily  justify  rejection  of  each  guideline.  For 
example,  the  recommendation  to  train  under  mild  speed  stress 
is  contradicted  by  several  experiments  showing  that  correct 
responses  should  be  followed  by  increases  in  stimulus  display 
duration.  However,  these  experiments  were  not  explicitly 
designed  to  test  the  effectiveness  of  speed  stress  during 
training.  Certainly,  the  recommendation  is  sensible  if  the 
purpose  of  the  training  program  is  to  increase  the  speed  with 
which  a  student  performs  a  task.  Further  research  is 
required  to  determine  whether  or  not  speed  stress  during 
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training  is  useful  when  increasing  performance  speed  is  not  a 
training  objective. 

Only  one  experiment  contradicts  the  recommendation  to 
present  information  before  practice  (Lahey,  1981),  but  the 
validity  of  this  guideline  is  logically  questionable.  By 
definition,  computer-based  drills  and  simulations  emphasize 
practice.  Often,  the  presentation  of  information  is  simul¬ 
taneous  with  practice.  Only  tutorials  may  require  informa¬ 
tion  to  be  presented  separate  from  practice.  In  addition  to 
the  experiment  by  Lahey,  the  demonstrated  effectiveness  of 
drills  and  simulations  contradicts  the  recommendation  to 
present  information  before  practice. 

The  recommendation  to  randomize  the  sequence  of  instruc¬ 
tional  material  is  contradicted  by  other  recommendations 
about  sequence  and  by  the  results  of  Barsam  and  Simutis 
(1984)  .  Randomization  can  occur  at  several  levels,  and  at 
some  levels  it  may  be  necessary  for  effective  instruction. 

The  response  choices  in  a  multiple-choice  question  can  and 
should  be  randomized,  and  ordinarily,  the  order  of  items  in  a 
test  should  be  randomized.  However,  there  are  some  types  of 
instructional  material  that  simply  cannot  be  learned  if  the 
material  is  presented  in  a  random  manner.  When  sequence  of 
instruction  is  a  critical  training  variable,  the  most 
effective  sequence  must  be  determined. 

The  general  consensus  about  whether  part-task  training 
should  precede  whole-task  training  is  that  it  depends  upon 
the  partitioning  strategy  for  the  particular  task  (Wightman  & 
Lintern,  1985) .  However,  Wightman  and  Lintern  note  that  the 
effects  of  the  segmentation  strategy  may  be  confounded  with 
the  backward  chaining  technique  used  in  most  of  the  training 
effectiveness  research.  Further  research  is  required  to 
separate  the  effects  of  these  two  variables. 

Finally,  the  contradictions  in  recommendations  about  the 
presentation  of  reviews  may  be  resolved  if  designers  consider 
that  there  are  several  different  kinds  of  reviews.  Instruc¬ 
tional  material  can  be  reviewed  at  the  end  of  lengthy 
instructional  segments  prior  to  questioning  or  ending  a 
session.  Reviews  may  follow  incorrect  responses  during  a 
drill  or  test.  Questions  or  items  in  a  drill  may  be 
reviewed.  Whether  or  not  students  should  control  the 
presentation  of  reviews  may  depend  upon  which  type  of  review 
one  is  considering. 
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The  remaining  guidelines  in  Table  1  have  been  insuffi¬ 
ciently  addressed  by  empirical  research.  The  next  two 
sections  discuss  the  guidelines  that  are  not  contradicted  by 
any  other  guideline  (guidelines  14-27) ,  and  those  that  are 
contradicted  by  another  guideline  (guidelines  28-57)  . 

No  disagreement  among  authors.  Research  has  not  been 
located  to  compare  strategies  of  highlighting,  but  there  is 
no  reason  to  assume  that  the  use  of  graphic  highlighting 
would  be  any  less  effective  than  other  methods,  such  as 
capitalizing  and  underlining.  Computer  display  design  has 
been  studied  in  other  settings  (Brown,  1988) ,  and  computer- 
based  instructional  designers  might  benefit  from  the  results 
of  that  research. 

It  would  be  simple  enough  to  test  the  recommendation  to 
place  graphic  information  on  the  same  screen  as,  and  to  the 
left  of,  text.  However,  it  seems  doubtful  that  any  distinc¬ 
tion  between  right  and  left  brain  specificity  of  visual 
processing  would  result  in  performance  differences  during  CBI 
that  have  practical  significance.  When  a  student  attends  to 
a  visual  stimulus,  such  as  text  or  graphics  on  a  computer 
screen,  that  stimulation  impinges  on  receptors  in  the  fovea. 
The  fovea  must  shift  so  frequently  during  inspection  of 
stimuli  on  the  screen  that  any  spatial  differences  in  place¬ 
ment  of  text  and  graphics  may  be  irrelevant  for  training 
effectiveness . 

The  assertion  that  high  quality  graphics  are  unnecessary 
has  not  been  adequately  tested.  This  is  an  important  area 
for  research,  because  high  quality  graphics  may  be  more 
expensive  to  produce  than  low  quality  graphics.  The  quality 
of  graphics  required  for  effective  training  may  be  an  issue 
of  fidelity.  The  general  consensus  is  that  maximum  physical 
fidelity  (between  a  computer  graphic  and  the  real-world 
stimulus  that  the  graphic  represents)  is  not  always  necessary 
for  effective  training.  However,  it  is  inappropriate  to  say 
that  high  quality  graphics  are  never  necessary.  It  seems 
reasonable  to  assume  that  different  types  of  images  used  for 
different  purposes  in  training  require  different  levels  of 
graphic  quality.  Further  research  is  required  to  elucidate 
the  relationship  among  types  of  images,  training  functions  of 
images,  and  quality  of  graphics. 

Similarly,  animated  graphics  are  often  more  expensive  to 
produce  than  static  graphics.  The  results  of  McDonald  and 
Whitehill  (1987)  suggest  that  animation  is  required  for 
effective  training  in  the  perception  of  dynamic  visual 


stimuli.  However,  it  is  not  clear  whether  animation  is  worth 
the  cost  for  many  other  applications  of  CBI . 

Application,  discrimination,  and  rhetorical  questions 
are  sensible  alternatives  to  simple  recall  questions. 

Research  has  shown  that  application  and  discrimination 
training  are  effective  approaches  in  CBI.  It  is  likely  that 
application  and  discrimination  questions  would  enhance 
training  similarly.  Rhetorical  questions  may  serve  as  useful 
stimuli  in  a  training  program.  However,  by  definition,  it 
would  be  impossible  to  record,  evaluate,  and  remediate  a 
response  to  such  a  stimulus. 

Although  one  experiment  showed  no  differences  between 
tutorials  with  different  f rame-to-question  ratios  (Schloss  et 
al.,  1985),  these  results  are  surprising,  and  the  recommenda¬ 
tion  to  present  questions  frequently  should  be  researched 
further.  Frequent  presentation  of  questions  facilitates  more 
interaction  with  the  training  device.  If  interactivity  is  a 
critical  aspect  of  training,  it  is  reasonable  to  assume  that 
more  frequent  questions  would  be  effective. 

No  one  disagrees  that  questions  should  be  presented 
after  the  text  passages  to  which  they  refer,  assuming  that 
the  recommendation  does  not  refer  to  prelesson  questions.  It 
seems  sensible  to  evaluate  or  reinforce  students'  knowledge 
of  a  fact  only  after  they  have  been  presented  with  that  fact . 
Prelesson  questions,  however,  have  been  shown  to  enhance 
training  effectiveness  (Dalton  et  al.,  1988).  Although 
further  research  is  required,  it  is  likely  that  intralesson 
questions  presented  prior  to  a  text  passage  are  also 
effective . 

Although  this  guideline  has  not  been  tested,  it  seems 
sensible  to  permit  students  to  change  answers  at  any  point 
during  a  computer-based  quiz.  The  only  exceptions  might  be 
programs  designed  to  train  students  to  answer  questions 
rapidly  under  time  pressure. 

No  one  has  contradicted  the  recommendation  to  permit 
students  to  control  the  sequence  of  questions,  and  no 
research  has  been  located  to  address  that  recommendation.  If 
the  sequence  of  questions  is  unimportant  for  training  effec¬ 
tiveness,  there  is  no  reason  why  students  should  be 
restricted  from  modifying  it.  However,  further  research  is 
required  to  elucidate  the  effects  of  various  sequential 
strategies  before  declaring  that  sequence  is  an  unimportant 
variable . 
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The  guideline  that  feedback  should  be  specific  to  the 
type  of  error  has  been  evaluated  in  experiments  with  mixed 
results.  If  different  types  of  potential  errors  can  be 
identified  during  instructional  design,  it  would  be  possible 
to  program  different  types  of  feedback.  However,  the  added 
cost  of  such  an  effort  justifies  further  research  to 
determine  the  level  of  specificity  required  in  feedback. 

No  research  was  located  on  the  effects  of  novel,  enter¬ 
taining,  or  auditory  stimuli  in  error  feedback.  A  stimulus 
that  is  novel  or  entertaining  to  a  designer  might  not  be  so 
to  a  student.  Furthermore,  students  of  different  ages  may 
respond  to  such  stimuli  in  different  ways.  Presumably,  the 
recommendation  to  avoid  auditory  stimuli  is  made  to  prevent  a 
student's  embarrassment  when  making  an  error  in  a  CBI  class¬ 
room.  However,  many  CBI  programs  present  auditory  stimuli  as 
a  part  of  the  training.  Headphones  may  be  used  to  prevent 
the  stimuli  at  one  workstation  from  disrupting  the  training 
at  nearby  workstations.  Under  such  conditions  of  privacy, 
there  is  no  reason  to  assume  that  auditory  stimuli  would  be 
any  less  effective  than  visual  stimuli  for  remediating 
errors . 

The  guideline  to  place  feedback  on  the  same  screen  as 
the  question  assumes  that  such  feedback  always  fits  on  one 
screen.  If  feedback  is  extensi. /e,  it  may  be  necessary  to 
present  it  in  several  screens . 

Some  authors  disagree.  Authors  of  instructional  design 
guidelines  contradict  each  other  in  their  recommendations 
about  the  presentation  of  lesson  objectives  and  orienting 
instructions.  Guidelines  about  orienting  instructions  seem 
self-evident.  It  cannot  be  denied  that  CBI  using  novel 
equipment  or  complicated  interactivity  routines  must  provide 
instructions  on  how  to  use  the  equipment  or  interact  with  the 
program  properly.  When  the  equipment  is  simple  to  operate  or 
the  interactivity  routines  are  obvious,  orienting  instruc¬ 
tions  are  not  required.  Because  of  research  with  other 
educational  media,  some  authors  suggest  that  lesson  objec¬ 
tives  are  ineffective  or  may  limit  the  amount  of  material 
retained  from  CBI .  Other  authors  recommend  that  lesson 
objectives  (and  orienting  instructions)  be  presented,  at 
least  as  a  student-controlled  option.  These  assertions  have 
not  been  tested  with  CBI.  If  lesson  objectives  are  ineffec¬ 
tive  or  detrimental  to  training  effectiveness,  they  should 
not  be  presented.  Further  research  is  required  to  evaluate 
these  guidelines. 

The  recommendations  to  permit  students  to  control  the 
duration  of  questions  and  graphics  are  contradicted  by 
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recommendations  to  train  under  speed  stress  or  to  program  the 
computer  to  control  stimulus  display  durations.  Visual 
stimuli  should  be  presented  long  enough  for  students  to 
adequately  view  them.  The  best  strategy  for  determining 
stimulus  duration  probably  depends  upon  the  type  of  training 
involved. 

It  makes  sense  to  shorten  tasks,  if  this  can  be  done 
without  losing  important  aspects  for  training.  For  most 
tasks,  more  training  means  more  learning.  More  trials  can  be 
presented  if  unnecessary  time-consuming  events  are  eliminated 
from  the  training.  Designers  must  empirically  determine 
which  events  are  important  and  which  are  unnecessary  for  the 
specific  tasks  to  be  trained. 

Guidelines  36-40  in  Table  1  are  about  sequence  of 
instructional  material.  Authors  of  design  guidelines  dis¬ 
agree  about  whether  students  should  control  the  sequence  of 
instruction.  Research  must  be  conducted  to  determine  if 
sequencing  is  an  important  variable  for  training  effective¬ 
ness  and  what  sequence  is  most  effective  for  particular 
applications.  If  sequence  is  an  important  training  variable, 
students  should  not  be  permitted  to  control  it.  Students  are 
no  better  than  trained  instructional  designers  (and  probably 
worse)  at  determining  the  most  effective  sequences  of 
instruction.  If  an  effective  sequence  can  be  determined, 
that  sequence  should  be  presented.  Whether  or  not  any 
particular  sequence  (such  as  concrete  material  before 
abstract  material,  or  a  chronological  sequence)  is  effective 
depends  upon  the  nature  of  the  instructional  material.  Most 
material  can  be  presented  in  several  different  sequences. 

The  design  of  CBI  must  include  formative  evaluations  to 
determine  the  most  effective  sequences. 

Sequencing  instructional  material  according  to  diffi¬ 
culty  level  is  a  related  issue  (guidelines  41-45) .  For  most 
material,  simple  material  may  serve  as  a  prerequisite  for 
more  difficult  material.  In  such  cases,  the  prerequisite 
material  should  be  presented  first.  Research  is  required  to 
determine  when  the  difficulty  level  should  be  raised  during 
CBI  for  each  application.  Again,  it  makes  no  sense  to  depend 
upon  students  to  determine  the  most  effective  level  of  diffi¬ 
culty  for  training.  However,  it  is  possible  that  students 
using  CBI  for  skill  sustainment  training  could  effectively 
select  their  own  difficulty  level.  This  possibility  should 
be  evaluated  in  empirical  research. 

One  experiment  has  shown  that  limited  prelesson  ques¬ 
tioning  enhances  learning  and  extensive  prelesson  questioning 
could  be  detrimental  (Dalton  et  al.,  1988) .  However,  similar 
to  the  assertion  about  lesson  objectives,  the  assertion  that 
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prelesson  questions  limit  learning  requires  further  research 
(guideline  46) . 


The  recommendation  to  permit  students  to  control  the 
number  of  problems  (guideline  47)  is  contradicted  by  the 
empirically  supported  recommendation  to  program  the  computer 
to  control  adaptively  the  number  of  problems.  In  fact,  the 
research  shows  no  differences  between  adaptive  computer 
control  and  student  control  when  students  are  advised  about 
the  number  of  problems  required  to  master  the  material. 
Adaptively  generated  advice  appears  to  be  a  necessary 
component  of  the  effective  student-control  strategy.  If 
students  follow  such  advice,  there  is  no  reason  to  prevent 
them  from  controlling  the  number  of  problems.  Whether  or  not 
students  will  follow  the  advice  is  a  topic  for  further 
research . 

Guidelines  48-51  in  Table  1  address  the  importance  of  a 
student's  response  in  CBI .  If  the  interaction  between  stu¬ 
dent  and  computer  is  an  important  phenomenon  in  training,  the 
nature  of  the  student's  response  is  important.  Permitting 
students  to  see  answers  before  attempting  to  answer  questions 
themselves  is  one  form  of  interaction.  Whether  this  type  of 
interaction  is  as  effective  as  requiring  an  overt  response 
and  then  providing  feedback  must  be  determined  in  empirical 
research . 

The  focus  of  feedback  research  has  been  on  remediation 
of  errors.  Guidelines  for  designing  feedback  for  correct 
responses  have  not  been  evaluated  (guidelines  52-54) .  Cer¬ 
tainly,  students  should  be  informed  when  they  have  made  a 
correct  response.  Therefore,  it  is  erroneous  to  say  that 
feedback  for  correct  responses  is  unnecessary.  However, 
whether  this  feedback  should  be  brief  or  extensive  may  depend 
upon  other  training  issues.  Further  research  is  required  in 
this  area. 

Whether  feedback  should  be  presented  immediately  follow¬ 
ing  a  response  or  should  be  delayed  for  some  time  period 
probably  depends  upon  other  training  issues  (guidelines 
55-57)  .  Research  in  basic  learning  processes  suggests  that 
immediate  feedback  is  most  effective  for  most  training 
applications.  The  generalizability  of  this  conclusion  has 
not  been  sufficiently  explored. 


Conclusion 

The  implications  of  this  review  may  differ  for  the  CBI 
designer  and  the  educational  technology  researcher.  Both 


should  be  aware  of  the  contradictions  and  potential  limita¬ 
tions  of  the  recommendations  in  the  literature.  Research  can 
be  designed  to  address  the  current  gaps  in  our  knowledge  of 
computer-based  instructional  strategies;  in  the  meantime, 
designers  must  continue  to  develop  material  to  meet  current 
training  and  educational  requirements.  Instructional 
designers  should  recognize  when  a  particular  strategy  is 
being  applied  in  a  case  that  potentially  tests  the  generaliz- 
ability  of  a  recommendation  or  research  result.  Conse¬ 
quently,  formative  and  summative  evaluations  during  CBI 
development  will  contribute  to  the  body  of  knowledge  in 
instructional  design  strategies  (Gagne  et  al.,  1988). 
Educational  technology  researchers  should  identify  the 
important  dimensions  along  which  the  generalizability  of 
instructional  strategies  may  vary.  Several  have  been 
suggested  here,  but  a  clearer  exposition  requires  further 
research. 
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